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Abstract 

Through  its  “Global  Reach”,  the  Air  Force  operates  globally  and  in  environments  that  are 
potentially  dynamic,  competitive,  and  lethal.  “Information  Dominance”  has  assumed  a  central 
role  in  such  environments.  In  order  to  achieve  Information  Dominance,  the  Air  Force  must  have 
trained  and  skilled  personnel  able  to  perform  highly  intensive  activities  using  information 
technologies.  Therefore,  it  is  crucial  that  information  technology  training  be  able  to  deliver  the 
requisite  skills  personnel  need  to  attain  and  sustain  performance  on  these  systems. 

The  importance  of  training  gives  rise  to  the  need  to  determine  how  information 
technology  training  can  be  continually  improved  to  maximize  return  on  training  dollars  and 
produce  adequately  trained  personnel.  Evaluation  of  information  technology  training  provides 
such  an  avenue  by  providing  information  that  assesses  how  well  the  training  program  is  meeting 
its  goals,  both  during  training  and  on-the-job. 

This  thesis  analyzes  the  impact  of  information  technology  training  on  trainee  reactions, 
learning,  and  performance  via  an  information  technology  training  program  for  a  global  Air  Force 
command  and  control  (C2)  system.  Trainee  confidence,  perceptions,  attitudes,  test  scores,  and 
performance  with  regard  to  the  C2  system  are  analyzed  to  discover  if  the  training  positively 
increases  any  of  these  factors.  Among  the  most  important  results  presented  is  the  degree  of 
trainee  on-the-job  performance  improvement  using  the  system  following  training. 

With  training’s  relationship  to  these  factors  clearly  outlined  and  understood,  information 
technology  training  courses  can  be  improved  and  resources  allocated  to  those  courses  that 
consistently  produce  skilled  personnel  who  can  perform  using  information  technology. 
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AN  ANALYSIS  OF  INFORMATION  TECHNOLOGY  TRAINING 


EFFECTIVENESS:  THE  IMPACT  ON  TRAINEE 
REACTIONS,  LEARNING,  AND  PERFORMANCE 


I.  Introduction 


Chapter  Overview 

This  chapter  discusses  how  the  pervasiveness,  growth,  and  importance  of 
information  systems  in  the  workplace  mandate  the  need  for  personnel  training  on  these 
systems.  A  theoretical  argument  supporting  the  need  for  training  is  developed  by  linking 
the  issues  of  information  technology  investment,  training  effectiveness,  and  trainee 
performance  to  information  systems  training.  Next,  operational  definitions  of  training, 
reaction,  learning,  and  performance  are  listed  along  with  a  model  depicting  their 
theorized  interactions.  Then,  investigative  questions  are  identified  that  define  the  scope 
of  this  research.  Finally,  the  contributions  of  this  thesis  to  the  Air  Force  and  an  outline  of 
subsequent  chapters  are  presented. 

Background 

Increasing  globalization  of  the  world’s  economy  demands  that  firms  also  operate 
globally.  Information  and  information  systems  provide  the  analytic  and  communications 
power  firms  need  for  conducting  trade  and  managing  business  on  a  global  scale  (Laudon 
and  Laudon,  199:3).  Because  of  global  information  and  communications,  customers  can 
now  operate  in  a  24-hour  market.  Firms  who  exploit  this  phenomenon  gain  a  significant 
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competitive  advantage  while  those  who  do  not  find  themselves  consistently 
outmaneuvered  (Hammer  and  Champy,  1993:100).  Robust  information  and 
communications  are  now  the  “price  of  entry”  into  the  global  market.  This  is  not  only  true 
in  the  commercial  sector.  The  Air  Force  operates  globally  and  in  environments 
potentially  more  dynamic,  competitive,  and  lethal  than  those  of  business.  In  fact,  the 
same  information  and  communications  capabilities  that  the  business  world  requires  are 
required  by  the  Air  Force  at  an  equal  or  even  higher  level. 

Information  technology  (IT)  plays  an  important  role  in  organizations.  According 
to  the  July,  1997,  edition  of  Datamation,  the  combined  annual  IT  revenue  in  1996  for  the 
world’s  top  100  IT  vendors  was  $502  billion.  This  figure  does  not  include  the  human 
resources  invested  in  developing  and  running  information  technology  systems  in  all  the 
firms  that  use  IT  (as  opposed  to  inventing,  making,  and  selling  it).  Thus,  it  seems 
reasonable  to  conclude  that  1997  worldwide  expenditure  on  IT-related  activities  likely 
exceeds  $  1  trillion  per  annum  . 

The  tremendous  deployment  of  computerized  technologies  over  the  past  decade 
has  changed  the  face  of  the  workplace.  Sophisticated  computerized  technologies  are  now 
common  in  factories,  offices,  and  on  the  battlefield.  Based  on  census  numbers  between 
the  years  1984-1989,  the  number  of  workers  using  a  computer  at  work  increased  from 
24.6%  to  37.4%:  a  50%  increase  (Krueger,  1993).  Over  35%  of  firms  reported  they  had 
personal  computers  in  1989,  compared  with  fewer  than  10%  in  1984.  Experts  predict  that 
by  the  year  2010,  new  information  technologies  will  affect  the  jobs  of  approximately 
90%  of  the  workforce  (Hines,  1994).  In  fact,  a  1997  study  by  the  United  States 
Department  of  Commerce  Office  of  Technology  Policy  warned  of  a  growing  IT  staffing 
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shortage  as  information  technology  becomes  more  important  to  companies.  The  report 
estimates  that  the  United  States  will  require  more  than  1  million  IT  workers  by  2005,  and 
if  the  IT  worker  shortage  is  not  resolved  soon,  the  report  suggests  the  U.S.  economy  may 
suffer  a  significant  decline  in  productivity  across  all  business  sectors.  With  the  majority 
of  personnel  using  some  form  of  IT  in  their  jobs,  IT  training  becomes  a  vital  issue. 

The  facts  show  that  a  large  and  growing  proportion  of  the  world’s  economic 
resources  and  human  resources  are  now  devoted  to  the  development,  production,  and  use 
of  information  technologies  (Laudon  and  Laudon,  1997:4).  Organizations  invest  in 
information  technology  because  they  want  to  increase  productivity,  gain  a  competitive 
advantage,  reengineer  processes  to  become  more  responsive  to  customer  needs,  or 
provide  better  information  for  decision-making.  With  so  many  billions  of  dollars  being 
invested  in  IT  each  year  and  the  human  resources  required,  it  is  important  that 
organizations  understand  how  to  apply  the  technology  effectively.  One  way  to  be 
effective  is  to  train  personnel  to  use  IT. 

Training  is  defined  as  a  planned  effort  by  an  organization  to  facilitate  the  learning 
of  job-related  behavior  on  the  part  of  its  personnel  in  order  to  meet  organizational  goals 
(Wexley  and  Latham,  1991 :3).  Training  plays  a  crucial  role  in  the  successful 
implementation  of  computerized  technologies  in  the  workplace  and  in  improving 
competitiveness.  Ample  research  and  practical  experience  show  that  training  is  an 
effective  means  of  increasing  organizational  productivity  (Burke  and  Day,  1986;  Guzzo, 
Jette,  and  Katzell,  1985;  Russell,  Terborg,  and  Powers,  1985).  A  qualitative  study  by 
Klein  and  Ralls  (1995)  found  that  training  was  mentioned  as  an  essential  success  factor 
for  technology  implementation  in  67%  of  18  case  studies.  McKersie  and  Walton  (1991) 
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reported  that,  in  six  case  studies  of  technology  implementation,  interviewees  emphasized 
the  importance  of  training  in  facilitating  organization  change  and  successfully 
introducing  new  technology.  If  spending  on  training  is  any  measure,  firms  and 
government  organizations  seem  to  agree  that  training  is  important.  Commercial  firms 
spend  $30  billion  on  formal  training  and  another  $180  billion  on  on-the-job  training 
(Tannenbaum  and  Yukl,  1992). 

The  Air  Force  is  no  different.  According  to  the  President’s  budget  for  FY  1999, 
the  Air  Force  training  budget  is  projected  to  be  $1 .7  billion.  The  trend  toward  training  in 
general  and  IT  training  in  particular  is  not  projected  to  abate  any  time  soon.  In  fact,  the 
top  ten  human  resource  development  (HRD)  trends  all  center  around  training  and 
computer  training  in  particular,  according  to  the  Association  for  Training  and 
Development’s  July  1997  National  HRD  Executive  Survey.  These  trends,  in  order  of 
importance,  are  listed  below. 

Top  10  HRD  Trends 

1 .  Computer  skills  training 

2.  Teamwork  training 

3.  Shift  from  training  to  performance 

4.  Decision-making  and  problem-solving  training 

5.  Rapid  development  and  deployment  of  training 

6.  Systems-thinking  training 

7.  Demonstrating  training  outcomes 

8.  Measuring  performance  outcomes 

9.  Shift  from  training  to  learning 

10.  Making  a  business  case  for  training  interventions 

These  trends  amplify  the  fact  that  an  organization  that  wishes  to  continually 
leverage  its  IT  investment  must  ensure  that  it  has  competent  personnel  to  operate  and 
maintain  the  system.  This  link  between  information  systems  and  training  is  established 
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during  the  initial  deployment  of  the  system  and  is  sustained  until  the  end  of  its  life  cycle. 
It  makes  sense,  then,  to  view  IT  training  as  a  part  of  a  process-oriented  system  that  uses 
continual  evaluation  of  its  IT  training  programs  to  measure  program  effectiveness.  The 
information  gained  from  evaluation  can  help  determine  where  new  training  techniques 
can  be  incorporated  to  improve  training  effectiveness  and  maximize  return  on  training 
dollars  (Goldstein,  1974:24). 

The  necessary  evaluative  information  must  be  obtained  and  used  to  modify  and 
improve  IT  training  programs.  If  a  training  program  accomplishes  its  stated  objectives, 
then  it  can  be  considered  effective  (Quinones,  1997:179).  Systematic  training  evaluation 
helps  eliminate  subjective  judgments  of  training  effectiveness  that  often  lead  to  difficulty 
in  justifying  training  costs  and  to  training  budget  cuts  during  fiscal  downturns  (Zenger 
and  Hargis,  1982:  1 1).  Worse,  the  organization  may  lose  the  competitive  advantage 
gained  from  the  use  of  its  information  system  as  well  as  its  investment.  Brinkehoff 
(1991)  suggests  that  training  programs  be  evaluated  in  order  to  provide  convincing  data 
for  training  effectiveness  and  determine  their  contribution  to  an  organization's  goals. 
Specifically,  he  suggests  that  training  programs  should: 

-  be  aimed  at  important  and  worthwhile  organizational  benefits 

-  use  the  best  available  and  most  cost-effective  designs  and  plans 

-  operate  smoothly  and  efficiently  and  are  enjoyed  by  participants 

-  achieve  important  skill,  knowledge,  and  attitude  objectives 

-  be  used  effectively  on  the  job 

-  produce  valuable  and  cost-effective  organizational  benefits 
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In  military  terms,  training  should  be  a  "force  multiplier"  that  improves  mission 
effectiveness  with  fewer  resources  (Salas,  Cannon-Bowers,  Kozlowski,  1997:365). 

Despite  the  value  of  evaluating  training,  determining  IT  training  effectiveness  is 
not  easy.  Many  individual  and  situational  factors  influence  IT  training  effectiveness  such 
as  demographics,  abilities,  attitudes,  tools  and  equipment,  financial  and  budgetary 
support,  climate,  and  culture,  among  others  (Mathieu  and  Martineau,  1997:199-214). 
Fundamentally,  however,  managers  must  determine  if  individual  IT  training  transfers  to 
the  workplace  and  whether  it  has  impact  on  the  organization  (Compeau  and  Higgins, 
1995:24). 

A  good  evaluation  process  centers  around  two  procedures:  establishing  measures 
of  training  success  (criterion  measures)  and  using  experimental  and  nonexperimental 
designs  to  determine  if  desired  changes  have  taken  place  as  a  result  of  training 
(Goldstein,  1974:24).  At  a  minimum,  criteria  must  be  established  for  both  the  evaluation 
of  trainees  at  the  conclusion  of  the  training  program  and  the  evaluation  of  on-the-job 
performance. 

Research  Context 

Air  Mobility  Command  (AMC)  has  implemented  and  maintains  a  robust 
training  development  and  evaluation  program  centered  on  command  and  control  and 
information  system  administrator  personnel  required  to  utilitize  its  Command  and 
Control  Information  Processing  System  (C2IPS).  AMC  uses  an  instructional  system 
development  (ISD)  process  to  plan,  design,  develop,  and  implement  training 
programs  in  an  effective  and  cost-efficient  manner.  As  originally  adopted  in  the  Air 
Force,  the  goal  of  instructional  systems  development  was  to  increase  the  effectiveness 
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and  efficiency  of  education  and  training  by  fitting  instruction  to  jobs,  eliminating 
irrelevant  knowledge  while  ensuring  that  trainees  acquire  the  necessary  skills, 
knowledge,  and  attitudes  to  do  the  job  (AFMAN  36-2234, 1993:7).  ISD  requires  that 
training  design  meets  specific  job  requirements  that  have  been  identified  through 
training  needs  assessments,  that  training  is  designed  for  all  job  tasks  and  knowledge 
necessary  for  successful  performance  on  the  job,  and  that  training  is  designed  to  meet 
specific  training  objectives  (AMC/DOOC,  1997:7).  Instruction  is  then  provided  in 
the  areas  most  critical  to  job  performance  and  is  not  wasted  on  areas  having  a  low 
probability  of  meeting  immediate  or  critical  long-term  needs. 

The  training  evaluation  framework  chosen  to  evaluate  the  C2IPS  training  program 
was  first  developed  by  D.L.  Kirkpatrick  in  1959.  Today,  the  Kirkpatrick  framework  is 
used  by  67%  of  organizations  that  conduct  evaluations.  It  provides  the  study’s 
measurement  framework  (ASTD,  1997:3).  The  framework  includes  criteria  for  assessing 
trainee  reactions  to  training,  learning,  and  performance.  In  addition,  the  information 
technology  based  nature  of  the  C2IPS  training  program  is  addressed  through  the  use  of 
four  criteria  from  the  management  information  system  (MIS)  literature. 

Kirkpatrick  training  criteria  include  reaction,  learning,  behavior,  and  results. 
Reaction  criteria  measure  trainees’  affective  response  to  the  quality  or  relevancy  of 
training  (Kraiger  and  Jung,  1997:152).  Reaction  responses  constitute  the  opinions  or 
“feelings”  of  the  participants  and  are  usually  captured  through  questionnaires.  This  study 
uses  four  types  of  MIS  reaction  criteria  previously  studied  in  the  literature:  computer  self- 
efficacy  (Compeau  and  Higgins,  1995),  computer  attitude  (Igbaria  and  Parasuraman, 
1991),  perceived  usefulness  (Davis,  1989),  and  perceived  ease  of  use  (Davis,  1989). 
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Computer  self-efficacy  measures  trainee  confidence  in  the  ability  to  accomplish  relevant 
computer-related  job  tasks.  Computer  attitude  measures  trainee  attitudes  toward  using 
computers  in  accomplishing  relevant  computer-related  job  tasks  (Igbaria  and 
Parasuraman,  1991 :555).  Perceived  usefulness  measures  how  much  a  trainee  thinks  a 
given  computer  system  will  improve  his/her  job  performance  (Davis,  1989:985). 
Perceived  ease  of  use  measures  the  degree  to  which  a  trainee  expects  a  given  computer 
system  to  be  free  of  effort  (Davis,  1 989:985).  Learning  criteria  measure  the  learning  of 
principles,  facts,  and  techniques  specified  in  training  program  objectives  (Kraiger  and 
Jung,  1997: 1 52).  Learning  measures  must  be  “objective  and  quantifiable  indicants  of  the 
learning  that  has  taken  place  in  the  training  program”  (Goldstein,  1974:60).  Learning 
measures  are  obtained  through  academic  tests  administered  as  part  of  the  C2IPS  training 
courses.  Behavior  or  performance  criteria  capture  on-the-job  behavior  resulting  from  the 
training  program  (Kraiger  and  Jung,  1997:152).  They  are  measured  using  tailored 
questionnaires  given  to  the  trainee  and  the  trainee’s  supervisor.  Figure  1  depicts  the 
modified  Kirkpatrick  model  used  for  this  study. 


THE  INFORMATION  TECHNOLOGY 
TRAINING  EVALUATION  MODEL 
(adapted  from  Kirkpatrick,  1959) 


Figure  1.  The  Information  Technology  Training  Evaluation  Model 
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Research  Problem 


This  study  uses  Air  Mobility  Command's  Command  and  Control  Information 
Processing  System  (C2IPS)  training  program  as  a  platform  to  investigate  the  impact  of 
individual  C2IPS  training.  C2IPS  is  an  AMC  information  system  that  tracks  airlift, 
coordinates  tankers,  and  interfaces  with  other  command  and  control  systems.  The 
competitive  advantage  of  C2IPS  lies  in  its  ability  to  provide  real-time  visibility  and 
control  of  airlift  missions  for  units  below  the  MAJCOM  level.  For  C2IPS  to  achieve  its 
purpose  of  tracking  and  controlling  airlift,  personnel  must  be  effectively  trained  on  the 
system.  As  a  tool  in  evaluating  training  success,  program  managers  responsible  for  AMC 
C2IPS  training  expressed  a  desire  for  evaluative  data  on  the  impact  of  C2IPS  training  on 
on-the-job  C2IPS  performance. 

Investigative  Questions 

The  following  research  questions  form  the  basis  of  this  thesis. 

-  What  is  the  relationship  between  trainee  reactions,  learning,  and  performance? 
Does  information  technology  training  increase  trainee  reactions  and  learning? 

-  Does  information  technology  training  increase  performance  on  the 
information  systems  after  training? 

Answering  these  questions  will  shed  light  on  the  impact  information  technology 
training  has  on  trainee  reactions,  learning,  and  performance  and  allow  trainers  and 
supervisors  to  better  assess  the  impact  of  this  training  on  the  organization. 
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Thesis  Contribution 


This  thesis  contributes  to  the  understanding  of  information  technology  training 
and  how  it  impacts  performance.  For  Air  Mobility  Command  (AMC),  continuous  and 
successful  performance  of  C2IPS  tasks  are  vital  to  providing  command  and  control  of 
cargo  and  tanker  aircraft  in  support  of  “Global  Reach”  for  the  Air  Force.  The  results  of 
this  thesis  can  enable  AMC  and  Air  Education  and  Training  Command  (AETC) 
leadership  to  assess  and  improve  key  components  of  their  C2IPS  training  program  related 
to  C2IPS  operator  and  system  administrator  job  performance  as  well  as  efficiently 
allocate  resources  to  the  C2IPS  training.  Further,  the  Air  Force  Research  Laboratory 
(AFRL)  can  integrate  the  results  of  this  study  into  development  of  future  command  and 
control  and  logistics  information  systems  training  programs.  By  understanding  the 
components  of  performance  and  how  information  systems  contribute  to  performance, 
AFRL  can  research  and  develop  more  effective  information  systems.  At  an  individual 
level,  training  instructors  and  supervisors  can  use  the  results  of  this  study  to  evaluate  the 
benefits  of  the  C2IPS  program  to  improve  course  content,  evaluate  training  progress,  and 
assess  performance  improvement.  In  addition,  the  research  methodology  used  in  this 
study  can  be  used  to  institutionalize  information  technology  training  evaluation  into  all 
AMC,  AETC,  and  AFRL  training  programs.  Finally,  this  research  can  make  a 
contribution  to  the  overall  academic  information  systems  and  training  literature  by 
increasing  understanding  of  the  relationships  between  training,  information  technology, 
and  job  performance. 
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Thesis  Outline 


Chapter  II  of  this  paper  will  present  a  review  of  applicable  literature  from 
academic  sources.  Chapter  III  will  provide  the  methodology  to  answer  the  propositions 
outlined  in  Chapter  I.  Chapter  IV  presents  a  statistical  analysis  of  the  data  collected  from 
the  study.  Finally,  a  discussion  of  the  results  and  subsequent  conclusions  will  be 
included  in  Chapter  V. 
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II.  Literature  Review 


Chapter  Overview 

The  deep  penetration  of  information  technology  into  the  planning  and  operation 
activities  of  an  organization  requires  new  considerations  for  managing  the  use  of  IT 
throughout  the  organization  (Lee,  Kim,  and  Lee,  1995:190).  Although  much  research  has 
been  conducted  on  the  consequences  of  computerization  on  performance,  productivity, 
and  employee  satisfaction,  empirical  literature  paints  an  inconsistent  picture  of  how  much 
impact  computerization  has  on  these  factors  (Klein  and  Ralls,  1997:324).  However, 
some  evidence  suggests  that  IT  can  have  a  significant  positive  impact  on  an  organization, 
especially  on  job  task  performance,  if  the  firm  provides  adequate  education  and  training 
for  all  levels  of  end  users  (Nelson  and  Cheney,  1987:547-559).  Classic  models  of 
organizational  training  support  this  view  of  training,  but  some  have  said  that  training  "has 
generally  been  slow  to  integrate  theoretical  concepts  from  other  research  areas"  and  is 
"too  micro  in  its  orientation"  (Kozlowski  and  Salas,  1997:247-248).  This  chapter 
suggests  an  approach  to  alleviate  these  concerns  by  integrating  traditional  training 
concepts  with  those  derived  from  management  information  systems  (MIS)  research  and 
training  research  in  the  hope  of  presenting  a  better  way  of  evaluating  the  relationship  of 
IT  training  to  job  task  performance. 

This  chapter  examines  several  areas  of  interest.  First,  the  evolution  of  training  as 
an  organizational  discipline  sets  a  historical  background  for  the  training  context.  Next, 
integrated  system  design  (ISD)  of  training  programs  is  addressed  with  special  emphasis 
on  Air  Force  application  of  ISD  and  the  notion  of  training  evaluation  within  ISD.  Then, 
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the  Kirkpatrick  framework  for  training  evaluation  is  outlined  including  discussion  of  the 
model’s  four  criteria:  reaction,  learning,  behavior,  and  results  and  their  relevance  to 
evaluation. 

Following  the  discussion  of  the  training  perspective,  the  MIS  training  perspective 
is  addressed.  IT  constructs  involving  IT  acceptance,  attitude,  and  self-efficacy  are 
described  in  relation  to  IT  training.  In  addition,  the  MIS  constructs  are  linked  to  similar 
organizational  behavior  constructs  to  support  hypothesized  independent 
variable/dependent  variable  relationships. 

Finally,  approaches  to  training  research  are  addressed  including  issues  of  design, 
reliability,  validity,  and  survey  research.  Because  performance  ratings  are  crucial  to  the 
outcome  of  this  study,  particular  attention  is  paid  to  the  nuances  of  self-report  and 
supervisor  performance  rating. 

Training  and  Training  Evaluation 

Background.  In  the  first  review  of  modem  industrial  training,  McGehee  (1949) 
noted  that  the  dictionary  definition  of  training  was  relatively  narrow  and  was  related 
solely  to  processes  associated  with  gaining  proficiency  in  a  specific  skill  or  competence 
(Ford,  1997:2).  He  also  noted  that  the  amount  and  nature  of  research  was  small  and 
anecdotal  (Ford,  1997:2).  In  contrast  to  this  narrow  definition  of  training,  McGehee  saw 
the  setting  of  industrial  training  as 

much  broader  in  scope  with  training  programs  designed  to  induct  the  new  worker, 
improve  the  performance  of  experienced  workers  and  managers,  and  to  inform  the 
worker  concerning  basic  economics  and  to  counteract  collectivistic  ideology. 
(Ford,  1997:4) 
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Interestingly,  McGehee  cited  the  large  amount  of  military  research  on  training  as 
potentially  valuable  to  industrial  training  including  applying  psychological  principles  to 
training,  evaluating  training,  standardizing  training  through  job  analysis  and  development 
of  lesson  plans,  and  designing  equipment  for  the  operator  (Ford,  1997:4).  McGehee 
asserted  that  there  was  a  need  for  more  systematic  approaches  to  training  and  provided  a 
list  of  training  problems  that  required  the  development  of  new  techniques  and  methods. 
These  problems  included  who  to  train,  what  the  content  of  training  should  be,  what 
methods  are  to  be  used  in  training,  who  is  to  do  the  training,  and  how  the  outcomes  of 
training  activities  are  to  be  evaluated  (McGehee,  1949).  With  regard  to  evaluation, 
McGehee  (1949)  maintained  that  the  main  reasons  for  a  paucity  of  training  evaluations 
were  a  lack  of  statistical  research  and  skill  among  training  personnel  and  a  lack  of 
support  among  top  management  for  controlled  training  studies  (Ford,  1997:  5). 

A  second  review  of  the  training  literature  conducted  by  Campbell  (1971)  revealed 
that  many  of  the  issues  addressed  by  McGehee  were  close  to  fruition.  Of  particular  note 
was  the  contributions  of  military  training  and  instructional  psychology  literatures  on  the 
development  of  the  Instructional  Systems  Design  Model  for  linking  training  needs 
assessment  to  training  objectives,  design,  and  evaluation  (Ford,  1997:6).  However, 
problems  still  remained  including  lack  of  focus  on  learning  objectives,  the  lack  of 
theoretical  models  and  empirical  studies,  and  the  lack  of  factors  impacting  training 
effectiveness  (Ford,  1997:7). 

The  most  recent  review  completed  by  Tannenbaum  and  Yukl  (1992)  concentrated 
on  training  needs  assessment,  design,  selected  training  methods,  trainee  characteristics, 
pretraining  and  posttraining  environments,  and  evaluation.  Tannenbaum  and  Yukl 
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demonstrated  that  many  of  the  problems  Campbell  identified  had  been  addressed 
including  a  research  base  that  described  the  multiple  factors  that  can  impact  training 
effectiveness  including  trainee  characteristics,  training  design  principles,  and  work 
context  (Ford,  1997:7).  In  addition,  their  summation  of  training  research  showed  that 
training  cannot  be  perceived  as  an  isolated  event,  but  rather,  as  part  of  the  ongoing 
dynamic  and  changeable  processes  in  work  organizations  (Ford,  1997:7).  The  military, 
because  of  its  need  to  train  large  number  of  recruits  and  personnel  for  a  variety  of  jobs, 
was  perhaps  the  first  to  recognize  the  need  for  an  integrated  systems  view  of  training 
(Goldstien,  1993).  It  was  one  of  the  first  organizations  to  adopt  the  Instructional  Systems 
Design  Model  (ISD). 

Instructional  Systems  Design  Model.  The  instructional  systems  design  model  is  a 
rational  approach  in  which  the  design  of  training  programs  is  based  on  needs  assessment 
and  the  psychological  research  on  learning  and  transfer  (Dipboye,  1997:31).  The  model 
fuses  aspects  of  training  from  the  fields  of  industrial  and  personnel  psychology  and  is  an 
application  of  the  philosophy  of  human  resource  management  (HRM).  HRM  assumes 
that  organizational  effectiveness  can  be  improved  through  people-oriented  activities  such 
as  staffing,  training,  and  wage  and  salary  administration  (Bolman  and  Deal,  1991).  The 
HRM  function  is  managed  in  a  way  to  maximize  the  economic  utility  of  the  HRM 
programs,  such  as  training  (Murray  and  Dimick,  1978).  For  HRM  programs, 
organizational  goals  serve  as  the  standards  against  which  the  programs  are  evaluated 
(Beer  and  Spector,  1984),  and  the  job  is  used  as  the  building  block  of  the  organization 
(Ash,  Levine,  and  Sistrunk,  1983).  Therefore,  the  primary  strategy  for  improving 
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organizational  effectiveness  is  to  provide  a  good  fit  between  the  person's  knowledge, 
skills,  and  abilities  and  the  requirements  of  the  job. 

ISD  implements  the  rational  HRM  approach  (Dipboye,  1997:32).  According  to 
this  model,  effective  training  is  based  on  a  careful  needs  assessment,  is  implemented 
through  "precisely  controlled  learning  objectives  design  to  achieve  instructional 
objectives"  (Goldstein,  1991 :5 14),  and  is  evaluated  against  performance  criteria.  The 
usual  process  is  to  start  with  a  formal  job  analysis  to  determine  the  knowledge,  skills,  and 
abilities  required  of  personnel  and  the  criteria  for  measuring  their  performance.  This  is 
followed  by  an  evaluation.  The  widespread  acceptance  of  the  Air  Force  ISD  model,  as  a 
rigorous  approach  for  managing  the  training  process  is  shown  in  the  frequent  use  of  this 
model  in  both  the  evaluation  literature  and  in  instructional  practice  (Goldstein,  1993; 
Latham,  1988;  Wexley,  1984;  AFMAN  36-2234).  In  fact,  the  Air  Force  uses  the  ISD 
model  as  the  foundation  for  its  training  programs  (AFMAN  36-2234,  1993:5).  Figure  3 
below  shows  the  Air  Force  ISD  model. 


Air  Force 

Instructional  Systems  Design  (ISD)  Model 


Figure  2.  Air  Force  Instructional  Systems  Design  Model  (AFMAN  36-2234, 

1993:5) 
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The  Air  Force  ISD  process,  first  implemented  in  1965,  provides  a  systems 
approach  to  planning,  developing,  implementing,  and  managing  education  and  training 
programs.  The  goal  of  Air  Force  ISD  is  to  increase  the  effectiveness  and  cost-efficiency 
of  education  by: 

-  Developing  instruction  based  on  job  performance  requirements. 

-  Eliminating  irrelevant  skills  and  knowledge  instructions  for  courses. 

-  Ensuring  graduates  acquire  the  skills,  knowledge,  and  attitudes  to  do  the  job. 
The  product  of  the  ISD  process  is  a  total  quality  instructions  system  that  is  cost-effective 
and  produces  graduates  who  can  do  the  job  (AFI  36-2201 :7).  In  the  ISD  model  the  last 
step  in  the  training  cycle  is  crucial  and  consists  of  a  rigorous  evaluation  of  the  training 
criteria  identified  in  the  needs  assessment  phase  (Dipboye,  1997:34).  An  evaluation 
should  allow  an  identification  of  which  criteria  have  changed,  whether  these  changes  are 
the  result  of  the  training  program,  and  whether  the  same  changes  will  occur  in  future 
replications  of  the  program  with  different  participants  (Dipboye,  1997:34). 

Evaluation.  In  the  Air  Force  ISD  model,  evaluation  is  a  continuous  activity  that  is 
integrated  throughout  each  stage  of  ISD,  beginning  with  analysis  and  continuing 
throughout  the  life  cycle  of  the  system  (see  Figure  3).  For  the  purpose  of  this  study, 
however,  operational  evaluation  is  the  main  concern.  Operational  evaluation  is  the 
continuous  process  of  gathering  and  analyzing  internal  and  external  feedback  data  to 
ensure  that  the  system  continues  to  effectively  and  cost-effectively  produce  graduates 
who  meet  established  requirements  (AFMAN  36-2234, 1993:77).  An  operational 
evaluation  looks  for  strengths  and  weaknesses  in  on-going  training  and  determines  the 
following: 
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-  How  well  the  graduates  are  meeting  job  performance  requirements. 

-  Whether  instruction  is  being  provided  that  is  not  needed. 

-  Whether  any  needed  instruction  is  not  being  provided. 

-  How  well  each  system  component  is  contributing  to  overall  system  quality. 

-  Ways  to  improve  the  graduate's  performance  as  well  as  the  training  program. 
The  two  operational  evaluation  activities  are  internal  and  external  evaluation.  Internal 
evaluation  gathers  and  analyzes  internal  feedback  and  management  data  from  within  the 
training  environment  to  assess  the  effectiveness  and  quality  of  the  instructional  process. 
External  evaluation  gathers  and  analyzes  external  feedback  data  from  the  field  to  assess 
graduates'  on-the-job  performance  in  an  operational  environment.  In  the  Air  Force  ISD 
model,  evaluation  is  key  to  continual  success  in  achieving  training  program  goals.  For 
this  study  focus  is  on  graduate  performance. 

The  Kirkpatrick  Framework.  Like  ISD,  the  Kirkpatrick  training  evaluation 
framework  is  an  outgrowth  of  behavioral  psychology  and  general  systems  theory. 
Although  the  framework  is  not  specifically  identified  in  Air  Force  publications 
addressing  evaluation,  Kirkpatrick’s  evaluation  criteria  are  readily  applicable  to  the  Air 
Force  ISD  model  (AFMAN  36-2234:99-100).  From  this  standpoint,  the  Kirkpatrick 
framework  is  particularly  well  suited  to  evaluation  of  Air  Force  training  programs. 
Figure  4  shows  the  framework. 

Reaction,  aside  from  being  the  most  popular  measure  of  training  (Alliger, 
Tannenbaum,  Traver,  Bennett,  Shotland,  1997:341),  is  often  a  critical  factor  for  the 
continuance  of  training  programs  (Phillips,  1991 :44).  Reaction  measures  how 


18 


The  Kirkpatrick  Training  Evaluation  Framework 

(1959) 

(from  Alliger  et.  al.,  Personnel  Psychology.  1997) 


How  do  trainees  feel  about  the  course? 


What  knowledge  or  skills  did  the  trainee  learn? 


What  change  in  on-the-job  behavior  has  been 
realized? 


What  is  the  financial  impact  of  training  on  the 
organization? 


Figure  3.  The  Kirkpatrick  Training  Evaluation  Model  (1959) 


participants  feel  about  a  program  and  what  a  person  thought  of  the  training  program 
(Kirkpatrick,  1977:9).  Most  trainers  believe  that  initial  receptivity  provides  a  good 
atmosphere  for  absorbing  the  material  in  the  training  program  but  that  it  does  not 
necessarily  produce  high  levels  of  learning  or  performance  (Goldstein,  1974:60). 

Kirkpatrick’s  next  evaluation  criterion,  learning,  is  typically  indexed  by  results  of 
traditional  tests  of  declarative  knowledge  (Alliger,  et  al.,  1997:345)  and  is  intended  to 
provide  an  indication  of  how  well  trainees  have  mastered  the  knowledge  and  skills  taught 
to  them  during  a  training  course  (Kraiger  and  Jung,  1997:152).  Learning  may  be  defined 
as  the  knowledge  and  skills  a  trainee  has  acquired  as  a  result  of  training  (Kirkpatrick, 
1977:9).  Test  content  is  compiled  from  the  content  of  the  training  course,  and  multiple 
test  items  are  constructed  and  administered  to  trainees  at  the  end  of  or  throughout  the 
course  (Bretz  and  Thompsett,  1993).  Learning  is  said  to  have  taken  place  when  attitudes 
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are  changed,  knowledge  is  increased,  or  skill  is  improved  as  measured  by  the  test 
(Willyerd,  1997:53).  Tests  are  popular  because  they  are  easy  to  develop  and  administer, 
and  because  they  produce  variance  in  trainees’  scores  (Goldsmith  and  Kraiger,  1997:74). 

The  impact  of  learning  on  job  performance,  although  not  strong,  has  been 
demonstrated.  A  meta-analysis  of  34  studies  using  Kirkpatrick’s  learning  criteria 
revealed  a  moderate  correlation  between  three  categories  of  learning  and  on-the-job 
performance  (r’s  =.  08-.18;  Alliger  et  al.,  1997:349).  The  relationship  between  learning 
and  reaction  (utility  judgments)  is  also  moderate  (r’s  =  .02-.26;  Alliger  et  ah,  1997:349) 
These  correlations  indicate  that  although  a  trainee  may  score  well  on  a  test  of  learning, 
superior  training  performance  in  terms  of  learning  does  not  always  guarantee  comparable 
performance  on  the  job.  A  study  by  Severin  (1959)  found  that  training  did  not  always 
accurately  represent  performance  on-the-job  and  should  not  be  substituted  for  studies  of 
on-the-job  performance  without  first  determining  that  a  strong  relationship  exists 
(Goldstein,  1974:  60).  Nevertheless,  measures  of  learning  criteria  provide  a  means  of 
assessing  trainee  learning  throughout  a  training  course  and  provide  some  indicator  of  on- 
the-j  ob-performance . 

Behavior  is  Kirkpatrick’s  third  criteria  and  is  defined  as  the  extent  to  which  a 
person’s  behavior  changes  due  to  attending  training  (Willyerd,  1997:53;  Kirkpatrick, 
1977:9).  Although  training  programs  can  facilitate  knowledge  acquisition  (learning), 
there  may  be  a  large  gap  between  knowing  facts  or  principles  and  demonstrating  them  on 
the  job.  It  is  possible  for  trainees  to  do  well  on  performance  tests  administered  during 
training,  yet  not  be  able  or  willing  to  exhibit  these  same  skills  on  their  jobs.  In  other 
words,  there  is  no  transfer  of  learning  to  the  job  (Wexley  and  Latham,  1991:1 19).  If  the 
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purpose  of  a  training  program  is  to  produce  on-the-job  performance  improvement 
(Goldstein,  1974:3),  on-the-job  performance  criteria  are  vital  to  measuring  performance 
improvement  after  training,  or  transfer  of  training.  Typically,  transfer  is  assessed  by 
measuring  the  maintenance  and  generalization  of  trained  skills  after  the  trainee  has  been 
on  the  job  for  some  time  (Baldwin  and  Ford,  1988).  In  the  Air  Force  ISD  model, 
operational  evaluation  of  on-the-job  performance  after  training  is  the  critical  element  for 
assessing  the  effectiveness  of  training  programs  (AFMAN  36-2234, 1993:104). 

Evaluation  of  results,  Kirpatrick’s  fourth  criteria,  can  be  stated  in  terms  of  cost- 
related  objectives  that  provide  a  dollar  value  of  training  (Wexley  and  Latham,  1991:120). 
Results  are  defined  as  the  impact  of  training  on  organizational  objectives  such  as  profits, 
production,  quality,  etc.  (Kirkpatrick,  1977:9).  The  concept  of  utility  is  synonymous  with 
results  evaluation  and  assesses  the  training  program  in  terms  of  its  cost  versus  its  benefits 
in  financial  terms  (Cascio,  1989).  A  utility  analysis  is  important  because  it  is  possible  for 
a  training  program  to  bring  about  favorable  reactions,  increase  trainee  learning,  change 
employee  behavior  on-the-job,  and  improve  cost-related  results.  It  is  also  possible  that 
such  a  training  program  may  still  not  be  worth  implementing  in  an  organization  because 
it  is  not  cost-effective.  Due  to  the  focus  of  this  study,  results  criteria  were  not  examined. 

Based  on  the  above  discussion,  the  following  propositions  are  advanced: 

PI :  Information  technology  training  increases  a  trainee’s  reactions,  learning,  and 
performance. 

P2:  The  better  a  trainee’s  reactions  to  IT  training,  the  higher  a  trainee’s  learning. 

P3:  The  better  a  trainee’s  reactions  to  IT  training,  the  better  a  trainee’s 
performance. 
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The  value  and  impact  of  the  Kirkpatrick  training  evaluation  model  can  not  be 
understated.  Its  pervasive  use  in  industry  makes  it  a  practical  choice  for  evaluating 
training  programs.  Despite  this,  little  work  has  been  done  to  evaluate  information 
technology  training  courses  with  validated  MIS  constructs  using  the  Kirkpatrick 
framework.  To  further  develop  this  point,  the  next  section  will  describe  several  MIS 
constructs,  how  they  might  integrate  into  the  Kirkpatrick  framework,  and  what  results 
might  be  expected  when  evaluating  IT  training  programs  using  MIS  constructs. 

Modifying  the  Kirkpatrick  Framework  to  Accommodate  MIS  Constructs  and  IT 
Training 

It  is  a  commonly  held  axiom  that  user  training  is  a  key  element  in  MIS  success 
(Galletta,  Ahuja,  Hartman,  Teo,  and  Peace,  1995:70).  A  study  by  Nelson  and  Cheney 
(1987)  showed  that  IT  can  have  a  significant  positive  impact  on  an  individual’s  job 
performance  if  the  firm  provides  adequate  training.  User  training  has  been  found  to 
influence  attitudes  toward  an  information  system,  and  user  training  programs  are  also 
likely  to  increase  user  confidence  in  their  ability  to  master  and  use  computers  in  their 
work  (Igbaria,  Guimaraes,  and  Davis,  1995:92).  In  addition,  previous  MIS  researchers 
have  focused  on  user  attitudes,  behavior,  and  performance  as  ways  of  measuring  IS 
success.  Many  attitudinal  or  judgment  measures  have  been  proposed  for  IT  success  such 
as  user  satisfaction  (DeLone  and  McLean,  1992:60),  user  expectations  (Ginzberg, 

1981 :459),  user  motivation  (Olfman  and  Bostrom,  1991 :249),  and  user  ability  (Lee  et  al., 
1995:192).  All  have  found  varying  degrees  of  support  in  the  MIS  literature.  In  this 
study,  however,  particular  attention  will  be  paid  to  four  MIS  constructs  that  have  proven 
effective  in  measuring  the  relationship  of  IT  training  to  attitudes  toward  IT  and  prediction 
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of  IT  usage.  These  constructs  are  computer  self-efficacy,  perceived  usefulness,  perceived 
ease  of  use,  and  computer  attitude.  After  defining  each  measure  and  describing  possible 
interrelationships  among  them,  a  modified  Kirkpatrick  model  is  presented  that 
hypothesizes  probable  relationships  among  traditional  Kirkpatrick  training  criteria  and 
the  MIS  measures. 

Theoretical  Background.  Social  Cognitive  Theory,  as  first  espoused  by  Bandura 
(1977)  advances  two  sets  of  expectations  as  major  cognitive  forces  guiding  behavior.  The 
first  set  of  expectations  relates  to  outcomes.  Individuals  are  more  likely  to  undertake 
behaviors  they  believe  will  result  in  valued  outcomes.  The  second  set  of  expectations 
encompasses  what  Bandura  (1977)  calls  self-efficacy,  or  beliefs  about  one’s  ability  to 
perform  a  particular  behavior.  Self-efficacy  influences  choices  about  which  behaviors  to 
undertake,  the  effort  and  persistence  exerted  in  the  face  of  obstacles  to  the  performance 
of  those  behaviors,  and  ultimately  the  mastery  of  those  behaviors. 

Computer  Self-Efficacy.  Computer  self-efficacy,  a  judgment  of  one’s  capability 
to  use  a  computer,  derives  its  theoretical  foundation  from  Albert  Bandura’s  self-efficacy 
work  (Murphy,  Coover,  and  Owen,  1989:893).  The  self-efficacy  construct  is  well- 
established  and  is  described  as  “judgments  about  how  well  one  can  organize  and  execute 
courses  of  action  required  to  deal  with  prospective  situations  containing  many 
ambiguous,  unpredictable,  and  often  stressful  elements”  (Bandura  and  Schunk, 

1 98 1 :587).  Self-efficacy  perceptions  have  been  found  to  influence  decisions  about  what 
behaviors  to  undertake  (Bandura,  Adams,  and  Beyer,  1977),  the  effort  exerted  and 
persistence  in  attempting  those  behaviors  (Brown  and  Inouye,  1978),  the  emotional 
responses  of  the  individual  performing  the  behaviors  (Stumpf,  Brief,  and  Hartman,  1987), 
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and  the  actual  performance  attainments  of  the  individual  with  respect  to  the  behavior 
(Locke,  Frederick,  Lee,  and  Bobko,  1984;  Wood  and  Bandura,  1989).  From  an  MIS 
perspective,  several  studies  have  examined  the  relationship  between  self-efficacy  with 
respect  to  using  computers  and  a  variety  of  computer  behaviors  (e.g.  Burkhardt  and 
Brass,  1990;  Hill  et  al,  1987).  These  and  other  studies  found  a  relationship  between  self- 
efficacy  and  performance  in  software  training  (Gist,  Schwoerer,  and  Rosen,  1989; 
Webster  and  Martocchio,  1992;  1993)  as  well  as  learning  (Martocchio,  1994:824).  Self- 
efficacy  measures,  including  computer  self-efficacy,  provide  the  strongest  predictive 
results  when  applied  to  specific  activities  such  as  education  or  skill  training  (Murphy  et 
al.,  1989:894).  Computer  self-efficacy  measures  can  then  be  used,  for  instance,  to  predict 
and  evaluate  performance  in  training  (Murphy  et  al.,  1989:894).  Using  C2IPS  as  the 
target  system,  this  leads  to  the  following  hypothesis. 

HI :  There  will  be  a  significant  positive  increase  in  a  trainee’s  computer  self- 
efficacy  following  C2IPS  training. 

H2:  There  will  be  a  significant  positive  relationship  between  a  trainee’s  computer 
self-efficacy  and  trainee  learning  on  C2IPS. 

H3:  There  will  be  a  significant  positive  relationship  between  a  trainee’s  computer 
self-efficacy  and  trainee  performance  on  C2IPS. 

Perceived  Ease  of  Use.  In  his  developmental  work  on  the  Technology 
Acceptance  Model,  Davis  (1989)  linked  self-efficacy  to  a  “perceived  ease  of  use” 
construct  in  developing  a  model  to  explain  user  acceptance  of  IT.  He  claimed  that 
PEOU  is  related  to  self-efficacy  in  that  both  notions  capture  judgments  of  how  well 
someone  performs  the  actions  required  when  faced  with  a  particular  situation  (Davis, 
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1989:321).  Although  PEOU  is  related  to  self-efficacy,  the  construct  captures  more  than 
this.  Davis  (1989)  states  that  PEOU  also  captures  instrumentality  of  computer  use.  For 
example,  improvements  in  ease  of  use  may  save  effort,  enabling  a  person  to  accomplish 
more  work  for  the  same  effort  (Davis,  1989:987).  Davis  defines  perceived  ease  of  use  as 
the  degree  to  which  the  prospective  user  expects  the  target  system  to  be  free  of  effort 
(Davis,  Bagozzi,  and  Warshaw,  1989:987).  The  easier  a  system  is  to  interact  with,  the 
greater  should  be  the  user’s  sense  of  efficacy  regarding  his  or  her  ability  to  carry  out  the 
tasks  required  to  operate  the  system.  Applying  this  reasoning  to  information  technology 
training,  then,  perceived  ease  of  use  might  increase  a  trainee’s  sense  of  self-efficacy  as 
well  as  on-the-job  performance.  Some  preliminary  support  suggests  a  possible  training  to 
perceived  ease  of  use  relationship.  Igabaria  et  al.  (1995)  found  that  user  training 
correlated  significantly  with  perceived  ease  of  use  (r  =  .43,  p  >  .001).  From  these  results, 
it  seems  reasonable  to  expect  training  to  positively  impact  perceived  ease  of  use  and  a 
subsequent  increase  in  trainee  performance  on  the  information  system.  However,  little 
has  been  done  to  investigate  the  link  between  perceived  ease  of  use  and  learning.  This 
study  provides  such  an  opportunity.  Given  the  above  discussion,  the  following 
hypotheses  are  advanced. 

H4:  The  will  be  a  significant  positive  increase  in  a  trainee’s  perceived  ease  of  use 
of  C2IPS  following  C2IPS  training. 

H5:  There  will  be  a  significant  positive  relationship  between  a  trainee’s  perceived 
ease  of  use  of  C2IPS  and  trainee  learning  on  C2IPS. 

H6:  There  will  be  a  significant  positive  relationship  between  a  trainee’s  perceived 
ease  of  use  of  C2IPS  and  trainee  performance  on  C2IPS. 
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Perceived  Usefulness.  The  second  aspect  of  Social  Cognitive  Theory,  outcomes, 


has  been  examined  by  Davis  (1989).  It  is  linked  to  Fishbein  and  Aizen’s  Theory  of 
Reasoned  Action  (1975).  Davis  (1989)  regards  his  measure  of  perceived  usefulness  as  a 
measure  of  IT  success.  Perceived  usefulness  is  defined  as  the  prospective  user’s 
subjective  probability  that  using  a  specific  computer  system  will  increase  his  job 
performance  within  an  organizational  context  (Davis  et  al.,  1989:985).  Davis  states  that 
“Bandura’s  ‘outcome  judgment’  variable  is  similar  to  perceived  usefulness.”  In  other 
words,  individuals  are  more  likely  to  choose  behaviors  (using  a  computer)  they  believe 
will  result  in  valued  outcomes  (Davis,  1989:321).  Perceived  usefulness  was  originally 
defined  as  part  of  an  adaptation  of  Fishbein  and  Aizen’s  (1975)  Theory  of  Reasoned 
Action.  In  this  theory,  performance  of  a  specified  behavior  is  determined  by  the 
behavioral  intention  to  perform  the  behavior.  Intentions  are  determined  by  the  person’s 
attitude  and  subjective  norm  concerning  the  behavior  in  question.  Similarly,  perceived 
usefulness  is  a  perceptual  construct  and  is  hypothesized  to  impact  a  trainee’s  attitude 
toward  using  a  computer.  This,  in  turn,  influences  intentions  to  use  a  computer  and 
subsequent  computer  usage.  If  perceived  usefulness  predicts  computer  usage,  it  may  also 
predict  trainee  performance  after  receiving  training  on  the  system.  A  study  linking 
training  to  perceived  usefulness  by  Igabaria  et  al.  (1995)  found  that  user  training 
correlated  significantly  with  perceived  usefulness  (r  =  .28,  p  <  .001).  Given  these  results 
it  seems  reasonable  to  expect  that  training  may  positively  impact  other  MIS  success 
constructs  as  well  as  eventual  trainee  performance  using  the  information  system.  Given 
this  discussion,  the  following  hypotheses  are  proposed. 
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H7:  There  will  be  a  significant  positive  increase  in  a  trainee’s  perceived 
usefulness  of  C2IPS  following  C2IPS  training. 

H8:  There  will  be  a  significant  positive  relationship  between  a  trainee’s  perceived 
usefulness  of  C2IPS  and  trainee  learning. 

H9:  There  will  be  a  significant  positive  relationship  between  a  trainee’s  perceived 
usefulness  of  C2IPS  and  trainee  performance  on  C2IPS. 

Computer  Attitudes.  In  the  final  analysis,  all  of  the  constructs  described 
contribute  toward  the  formation  of  broad  attitudes  toward  using  computers.  It  follows 
then,  that  a  measure  of  overall  attitudes  toward  computers  should  relate  in  some  form  to 
these  constructs.  When  described  in  terms  of  the  Theory  of  Reasoned  Action,  computer 
attitudes  affect  users’  behavioral  intentions  which  affect  users’  actual  usage  of  computers 
(Rainer  and  Miller,  1996:94).  Also,  Davis  showed  that  the  sum  of  PEOU  and  PU 
correlated  significantly  with  changes  in  attitude  toward  using  a  computer  (Davis, 
1989:994).  It  seems  reasonable  to  expect  that  if  a  training  program  can  change  neutral  or 
negative  attitudes  toward  computers  into  positive  attitudes,  a  trainee’s  subsequent  on-the- 
job  performance  may  be  increased.  Igbaria  and  Parasuraman  (1991)  cite  evidence  that 
user  training  is  positively  related  to  user  computer  attitudes  (r  =  .28,  p  <  .001).  Although 
no  study  has  been  done  on  the  relationship  between  computer  attitude  and  on-the-job 
performance,  the  organizational  behavior  literature  is  replete  with  studies  relating  job 
attitudes  to  job  performance.  For  example,  Noe  and  Schmitt  (1986)  report  that  job 
attitudes  influence  learning  and  behavior  change  as  a  result  of  training.  In  addition, 
Iaffaldano  and  Muchinsky  (1985)  performed  a  meta-analysis  on  the  relationship  between 
job  satisfaction  (a  “job  attitude”)  and  job  performance.  They  found  a  positive  correlation 
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between  the  two  measures  (r  =.  17,  p  =  .05).  Petty,  McGee,  and  Cavender  (1984)  meta- 
analyzed  the  satisfaction  and  performance  literature  and  found  a  stronger  positive 
correlation  between  the  two  variables  (r  =  .31).  Although  job  attitudes  and  job 
satisfaction  are  broader  attitudinal  constructs  than  computer  attitude,  it  would  not  be 
unreasonable  to  expect  computer  attitudes  to  influence  on-the-job  performance  in  a 
fashion  similar  to  the  satisfaction-performance  relations  described  above.  Changes  in 
computer  attitudes  as  a  result  of  information  technology  training  may  be  a  good  predictor 
of  performance.  Therefore,  the  following  hypotheses  are  advanced. 

H10:  There  will  be  a  significant  positive  increase  in  a  trainee’s  computer  attitude 
following  C2IPS  training. 

HI  1 :  The  will  be  a  significant  positive  relationship  between  a  trainee’s  computer 
attitude  and  trainee  learning. 

HI 2:  There  will  be  a  significant  positive  relationship  between  a  trainee’s 
computer  attitude  and  trainee  performance  on  C2IPS. 

The  Relationship  of  MIS  Constructs  to  On-the-Job  Performance.  MIS  research 
has  focused  primarily  on  IT  acceptance  and  usage  (e.g.  Davis,  1989;  Lee,  et  ah,  1995; 
Igabaria,  et  ah,  1995;  Snead  and  Harrell,  1994;  Burton,  Chen,  Grover,  and  Stewart,  1993; 
Compeau  and  Higgins,  1995)  rather  than  on  on-the-job  performance  using  the  system. 
Usage  has  been  defined  in  various  ways,  but  most  measures  rely  on  self-reports  of  the 
amount  of  time  spent  on  a  computer  or  software  package  per  day  (Igbaria  et  ah,  1995; 
Lee,  et  al,  1995).  These  measures  of  usage  are  satisfactory  if  amount  of  time  spent  on  a 
computer  is  the  measure  of  IT  success.  However,  most  training  programs  have  in  mind  a 
behavioral  change  that  positively  impacts  an  employee’s  performance  (Noe  and  Schmitt, 
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1986:497).  Use  of  a  computer  does  not  necessarily  mean  that  an  individual  is  performing 
job  related  tasks  at  an  acceptable  level. 

Performance  has  been  characterized  in  many  ways:  objectively,  subjectively,  in 
terms  of  quality  and  quantity  of  work,  and  in  terms  of  self-reports,  supervisor  ratings,  and 
peer  ratings.  All  methods  have  their  drawbacks  (Ianffaldano  and  Muchinsky,  1985:252; 
Cascio,  1991 :73).  However,  behaviorally-based  measures  focused  on  performance  of 
computer  related  tasks  may  provide  an  adequate  link  between  IT  training  and  on-the-job 
performance.  Behaviorally-based  measures  of  computer  performance  provide  the  rater 
with  a  series  of  descriptive  statements  of  job-related  behavior.  The  task  of  the  rater  in 
this  situation  is  to  indicate  how  well  the  ratee  performs  the  behavior  in  question  (Cascio, 
1991:86). 

In  the  Air  Force,  training  programs  are  developed  around  job  performance 
requirements  through  the  process  of  job  and  task  analysis  (Air  Force  Manual  36-2234 
1993:35).  Task  performance  captures  the  performance  domain  that  differentiates  one  job 
from  another  but  excludes  important  performance  elements  common  to  most  jobs  (Van 
Scotter  and  Motowidlo,  1996:525).  This  study  focuses  on  the  “technical  core”  of  C2IPS 
job  performance,  or,  in  other  words,  task  performance  in  using  C2IPS.  Linking  MIS 
success  constructs  to  behaviorally-based  computer  performance  ratings  can  add  a  new 
dimension  to  this  relationship.  Figure  5  depicts  the  modified  Kirkpatrick  IT  training 
model  and  shows  in  pictorial  form  the  hypotheses  outlined  above. 

Supervisor  and  Self-Rating  of  Performance  for  Training.  Measuring  performance 
after  training  is  the  most  desirable  way  of  assessing  a  training  program’s 
effectiveness  (Ghodsian,  Bjork,  and  Benjamin,  1997:84).  One  common  method 
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THE  INFORMATION  TECHNOLOGY 
TRAINING  EVALUATION  MODEL 
(adapted  from  Kirkpatrick,  1959) 


Figure  4.  Modified  Kirkpatrick  Evaluation  Framework  for  IT  Training 

of  assessing  performance  in  organizations  is  performance  appraisal.  Performance 
appraisal  involves  two  distinct  processes:  (1)  observation  and  (2)  judgment  (Cascio, 
199:77).  Observation  processes  are  more  basic,  and  include  the  detection,  perception, 
and  recall  or  recognition  of  specific  behavioral  events.  Judgment  processes  include  the 
categorization,  integration,  and  evaluation  of  information  (Thornton  and  Zorich,  1980). 
Observation  and  judgment  represent  the  last  elements  of  a  three-part  sequence  of  job 
analysis,  performance  standards,  and  performance  appraisal.  Job  analysis  identifies  the 
components  of  a  particular  job,  performance  standards  translate  job  requirements  into 
levels  of  acceptable/unacceptable  performance,  and  performance  appraisal  describes  the 
job  relevant  strengths  and  weaknesses  of  each  individual  (Cascio,  1991:77). 

Job  analysis,  among  other  functions,  helps  determine  how  personnel  are  to  be 
trained  and  what  criteria  will  be  used  to  measure  job  performance  (Cascio,  1991:189). 
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Performance  ratings  must  be  based  on  a  systematic  job  analysis,  and  they  should  be 
behaviorally  based  in  order  to  meet  technical  standards  and  legal  precedents  (Nathan  and 
Cascio,  1986).  For  the  purpose  of  this  study,  job  analysis  is  important,  as  it  is  the 
foundation  for  both  C2IPS  training  content  and  the  performance  measure  used  to  detect 
C2IPS  training  effects. 

Given  the  importance  of  performance  appraisals  for  an  individual’s  career  and  in 
rating  increases  in  performance  due  to  training,  who  performs  the  appraisal  is  vital. 

Raters  must  have  direct  experience  with,  or  firsthand  knowledge  of  the  individual  to  be 
rated  (Cascio,  1991 :84).  Several  different  sources  of  performance  rating  have  been  used 
including  the  immediate  supervisor,  peers,  subordinates,  self-appraisal,  and  customer 
appraisal.  In  this  study,  supervisor  and  self-appraisal  are  used,  therefore  the  remainder  of 
this  discussion  will  focus  on  these  two  performance  appraisal  methods. 

Up  to  93%  of  appraisal  programs  make  the  immediate  supervisor  the  rater 
(Bemadin  and  Beatty,  1984).  The  main  reason  for  this  is  that  the  supervisor  is  assumed 
to  be  the  person  with  the  most  opportunity  to  observe  the  individual  and  his/her  changes 
in  performance.  In  addition,  the  supervisor  is  probably  best  able  to  evaluate  each 
subordinate’s  performance  in  light  of  the  organization’s  overall  objectives  (Cascio, 

1991 :78).  In  some  settings,  participants'  supervisors  may  be  more  familiar  with  other 
factors  that  could  produce  performance  improvements.  It’s  simple,  inexpensive,  and  fairly 
credible  because  the  information  comes  from  the  "horse's  mouth"-in  this  case,  the 
supervisors  of  people  who  received  the  training.  Research  has  shown  that  feedback  from 
supervisors  is  more  highly  related  to  performance  than  from  any  other  source  (Becker 
and  Klimoski,  1 989).  With  regard  to  human  resource  programs  in  organizations  (e.g. 
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training),  reliable  measurement  of  performance  is  often  a  main  indicator  of  effectiveness 
(Viswesvaran,  Ones,  and  Schmidt,  1996:558).  As  such,  it  is  important  to  have  an  idea  of 
the  reliability  of  these  ratings. 

Several  analyses  of  supervisor  ratings  have  been  reported.  A  study  by 
Viswesvaran,  et  al.  (1996)  reported  the  interrater  reliability  of  supervisory  ratings  and 
overall  job  performance  to  be  r  =  .52.  In  addition,  a  meta-analysis  by  Conway  and 
Huffcutt  (1997)  reported  a  relationship  of  r  =  .50.  These  correlations  indicate  a  fairly 
strong  relationship  between  supervisor  ratings  and  overall  job  performance.  Although 
higher  correlations  are  desirable,  supervisor  ratings  continue  to  be  the  prime  source  of 
performance  ratings.  They  are  utilized  in  this  study. 

In  some  cases,  supervisor  performance  appraisal  can  be  supplemented  by  self¬ 
appraisal.  Because  an  individual  is  no  doubt  familiar  with  the  tasks  of  the  job,  it  seems 
reasonable  to  let  an  individual  judge  his  or  her  own  job  performance.  Their  actions  have 
produced  the  performance  improvements,  so  they  should  have  some  idea  of  how  much 
improvement  has  been  made  as  a  result  of  their  training.  In  addition,  self-appraisals 
provide  an  accessible  source  of  individual  performance  data  (Hoffman,  Nathan,  and 
Holder,  1991 :605).  Also,  management  tends  to  find  such  reports  credible  because 
participants  are  at  the  center  of  the  improvement.  Participants'  input  can  be  obtained  by 
asking  the  following  questions: 

-  What  percent  of  the  improvement  can  be  attributed  to  the  application  of 
skills,  techniques,  or  knowledge  gained  in  the  training? 

What  is  the  basis  for  your  estimation? 

-  What  degree  of  confidence  do  you  have  in  your  estimation? 
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What  other  individuals  or  groups  could  make  an  estimation? 

-  What  other  factors  do  you  think  contributed  to  the  improvement? 

The  opportunity  to  participate  in  performance  appraisal,  especially  when  combined  with 
goal  setting,  should  improve  an  individual’s  motivation  and  performance  (Latham  and 
Locke,  1979;  Mento,  Steel,  and  Karran,  1987).  However,  comparisons  of  self-appraisals 
with  those  of  supervisors,  peers,  and  subordinates  suggest  that  self-appraisals  tend  to 
show  more  leniency,  less  variability,  more  bias,  and  less  agreement  with  the  judgments  of 
others  (Harris  and  Schaubroek,  1988:55).  To  some  extent,  these  disagreements  may  stem 
from  the  tendency  of  raters  to  base  their  ratings  on  different  aspects  of  job  performance 
or  to  weight  facets  of  job  performance  differently  (Steel  and  Ovalle,  1984:668).  Self¬ 
raters  place  emphasis  on  personal  skill  and  technical  competence  while  supervisors  stress 
output  and  results  criteria  (Zammuto,  London,  and  Rowland,  1982). 

With  regard  to  training,  Golembiewski  and  Billingsley  (1980)  report  three  types 
of  change  associated  with  pre-,  post-,  and  on-the-job  self-report  data:  gamma  change, 
beta  change,  and  alpha  change.  Gamma  change  shifts  the  frame  of  reference  of  the 
trainee  as  a  result  of  training  (Cascio,  1991 :407).  This  means  that  trainees  think  about  a 
behavior  in  a  totally  different  way  than  before  training.  Beta  changes  adjust  the 
measurement  scale  before  and  after  training  (Cascio,  1991 :407).  Whereas  performance 
on  a  task  may  have  been  classified  by  the  trainee  as  “above  average”  before  training,  after 
training  that  same  behavioral  performance  may  become  “average”.  Alpha  change  is  the 
actual  change  in  behavior  over  time  with  respect  to  a  stable  research  instrument  and  a 
constant  training  situation  (Cascio,  1991 :407). 
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A  technique  for  accounting  for  changes  other  than  alpha  changes  when  obtaining 
self-reports  has  been  developed  by  training  practitioners.  The  technique  is  called  “Pre- 
Then-Post  Testing”  and  is  designed  to  account  for  response-shift  bias  or  bias  associated 
with  trainees  overestimating  abilities,  skills,  and  so  on  (Mezoff,  1981 :57).  Mezoff  states 
that  conventional  pre-post  testing  tends  to  be  inaccurate  because  participants’  frame  of 
reference  is  often  different  at  the  beginning  and  end  of  training  (gamma  change  or 
response-shift  bias).  To  make  legitimate  pre-post  comparisons,  a  non-biased  “Pre-“ 
measure  is  obtained  after  the  training  by  asking  the  trainee  to  reflect  back  to  his/her  level 
of  functioning  prior  to  the  training  and  then  re-rate  him/herself  (Mezoff,  1 981 :59).  This 
response  is  called  the  “Then”  measure  and  is  used  to  evaluate  training  effects  in  lieu  of 
the  original  “Pre-“  measure.  The  “Pre-Then-Post”  technique  for  correcting  self-report 
bias  has  been  addressed  in  the  research  with  generally  positive  results  (e.g.  Howard, 

1980;  Howard,  Schmeck,  and  Bray,  1979).  In  addition,  its  ease  of  administration  and 
adaptation  for  survey  use  as  well  as  its  potential  for  legitimately  documenting  the  effects 
of  training  make  it  of  practical  use  when  collecting  self-reports.  It  is  used  in  this  study. 

In  addition,  other  research  shows  that  self-appraisals  may  be  given  more  validity 
if  certain  techniques  are  utilized  (Mabe  and  West,  1982:293).  First,  instead  of  asking 
individuals  to  rate  themselves  on  an  absolute  scale,  Mabe  and  West’s  findings  (1982) 
suggest  that  a  relative  scale  that  allows  them  to  compare  their  performance  with  that  of 
others  will  improve  data  quality.  This  technique  tends  to  yield  closer  agreement  between 
self-  and  supervisor  ratings  (Farh  and  Dobbins,  1989).  Second,  provide  multiple 
opportunities  for  skill  appraisal.  The  skill  may  improve  with  practice  (Cascio,  1991 :80). 
Third,  provide  reassurance  of  confidentiality  (Cascio,  1991 : 80).  Confidentiality  may 
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promote  honesty  in  rating.  Fourth,  allow  personnel  to  “readjust”  their  performance 
assessment  given  the  results  of  training.  A  person’s  frame  of  reference  may  change  as  a 
result  of  training  thereby  promoting  a  more  realistic  assessment  of  skills. 

Interestingly,  research  indicates  that  data  from  multiple  sources  are  desirable 
because  they  provide  a  complete  picture  of  the  individual’s  effect  on  others  (Wohlers  and 
London,  1989).  Credibility  seems  to  rise  when  supervisors'  estimates  are  combined  with 
participants'  estimates  and  when  a  confidence  level  is  factored  in  (Phillips,  1996).  Other 
studies  cite  advantages  such  as  enhanced  ability  to  observe  and  measure  various  job 
facets  (Borman,  1974;  Henderson,  1984),  greater  reliability,  fairness,  and  ratee 
acceptance  (Latham  and  Wexley,  1982). 

Conway  and  Huffcutt  (1997)  report  that  the  number  and  types  of  raters  can  affect 
the  quality  of  rating  systems.  They  state  that  “multiple  raters  are  often  available. .  .this 
means  that  quality  of  ratings  can  be  improved. . .”  (Conway  and  Huffcutt,  1997:346).  A 
central  thesis  is  that  different  raters  provide  a  unique  perspective  on  performance 
(Conway  and  Huffcutt,  1997:349).  However,  they  state  that  increasing  reliability 
increases  validity  only  to  the  extent  that  the  performance  measures  in  question  possess 
validity  (Conway  and  Huffcutt,  1997:346).  Therefore,  using  the  results  of  job  analysis 
and  targeting  specific  behaviors  in  the  rating  process  is  critical.  A  meta-analysis  by 
Harris  and  Schaubroek  (1988)  reported  a  correlation  of  r  =  .35  between  supervisor  and 
self  ratings  of  job  performance.  In  terms  of  mean  differences,  self-ratings  have  been 
found  to  be  consistently  higher  than  supervisory  ratings  (Harris  and  Schaubroek, 

1988:55;  Conway  and  Huffcutt,  1997:342-344;  Hoffman,  et  al.,  1991:610).  The  reason 
for  this  difference  has  been  attributed  to  an  egocentric  bias  in  self-ratings  of  performance 
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(Harris  and  Schaubroek,  1988:55;  Conway  and  Huffcutt,  1997:351).  Despite  this 
inherent  inflation  of  self-ratings  of  performance,  self-appraisals  may  prove  highly 
correlated  with  other  measures  of  performance  when  performance  measures  are  verifiable 
and  specific  (Farh,  Werbel,  and  Bedeian,  1988).  When  coupled  with  supervisory  ratings, 
they  may  add  credibility  to  the  overall  performance  assessment. 

The  ease  of  obtaining  supervisory  and  self-ratings  of  performance  for  research 
purposes  is  a  salient  reason  for  their  use.  More  importantly,  reliability  of  these  ratings 
must  be  taken  into  account  when  analyzing  performance  changes  as  a  result  of  training. 
However,  a  research  design  that  incorporates  valid  and  specific  measures  of  task 
performance  along  with  simple  interventions  designed  to  increase  self-report  reliability 
can  help  offset  inherent  deficiencies  in  such  ratings.  With  due  consideration  for  these 
factors,  supervisory  and  self-ratings  can  be  of  practical  benefit  when  evaluating  training. 

Summary 

Chapter  II  summarized  the  literature  relevant  to  this  study’s  hypotheses  and 
research  design.  The  insights  gathered  from  this  literature  review  will  be  integrated  and 
applied  in  the  following  chapter  of  this  report.  In  particular,  Chapter  III  will  describe 
how  the  constructs  described  above  are  operationalized  and  explain  the  research  design 
and  procedures  used  to  evaluate  them. 
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III.  Method 


Chapter  Overview 

This  chapter  describes  and  justifies  the  research  method  used  to  test  the 
hypotheses  presented  in  Chapter  II.  First,  the  research  design  and  method  are  outlined. 
Next,  the  independent  variables  used  in  this  study  are  operationalized.  Procedures  for 
data  collection,  analysis,  and  measurement  are  also  presented,  including  the  results  of  a 
pilot  test. 

Research  Design 

Research  design  is  a  plan  for  selecting  the  sources  and  types  of  information  used 
to  answer  the  primary  research  questions  (Cooper  and  Emery,  1995:1 14).  The  design 
sets  the  framework  for  the  relationships  between  the  variables  of  the  study  and  outlines 
procedures  beginning  with  the  hypotheses  and  concluding  with  analysis  of  the  data 
gathered  (Cooper  and  Emery,  1995:1 14). 

Once  training  evaluation  criteria  are  identified,  a  successful  evaluation  hinges  on 
a  choice  of  appropriate  design  for  assessing  the  training  program  (Goldstein,  1974:24). 
For  training  evaluation,  appropriate  designs  follow  from  clear  elucidation  of  the  purpose 
of  the  evaluation,  what  is  to  be  evaluated,  and  how  the  evaluation  is  to  be  performed. 
Rigorous  evaluation  designs  are  essential  to  avoid  incomplete  evaluation,  invalid  results, 
or  simplistic  interpretation  of  results  (Newstrom,  1978:22).  Characteristics  of  a  good 
design  include  reliability,  validity,  and  relevancy  and  must  be  built  into  the  design  to 
ensure  credibility  for  effective  management  decision-making  (Salinger  and  Deming, 
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1982:20).  Campbell  and  Stanley  (1963)  describe  three  types  of  designs  and  their 
concomitant  reliability  and  validity  including  pre-experimental  designs,  true  experimental 
designs,  and  quasi-experimental  designs.  Although  true  experimental  designs  provide  the 
best  overall  validity  (Campbell  and  Stanley,  1963:8),  organizational  constraints  may 
preclude  such  designs.  Quasi-experimental  designs  provide  the  best  chance  for  achieving 
reliable  and  valid  results  while  accommodating  the  demands  of  the  organizational 
situation. 

Since  the  nature  of  military  work  precludes  random  assignment  of  subjects  and, 
hence,  a  true  experimental  design  and  because  of  the  difficulty  associated  with 
coordinating  three  separate  C2IPS  training  courses,  the  use  of  a  quasi-experimental 
design  is  the  best  option  for  this  study.  Over  the  period  of  the  study,  multiple  classes 
were  taught  in  each  course,  some  simultaneously,  at  different  locations.  Given  the 
researchers  other  duties  and  lack  of  funds,  it  was  deemed  impractical  to  travel  to  each 
training  class  to  conduct  experiments.  In  addition,  a  limited  trainee  population,  a  finite 
number  of  training  courses,  and  time  constraints  were  also  factors  contributing  to  the  use 
of  a  quasi-experimental  design.  The  quasi-experimental  design  used  in  this  study  can  be 
categorized  as  a  variation  of  a  time-series  quasi-experimental  design.  It  is  shown  in 
figure  6.  Data  were  collected  longitudinally  a  total  of  three  times  for  each  C2IPS  training 
class:  once  at  the  beginning  of  the  class,  once  at  the  end  of  the  class,  and  once  on-the-job. 

The  advantage  of  a  longitudinal  study  is  that  it  can  track  changes  over  time 
(Cooper  and  Emery,  1995:1 16).  For  this  study,  measurement  of  multiple  training  classes 
serves  to  overcome  some  of  the  validity  problems  associated  with  the  relatively  small 
number  of  measurements  over  time.  If  results  are  consistent  across  the  sample 
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C2IPS  Training  Research  Design* 


Beginning  of  Class  End  of  Class  On-the-Job 


d3*» 


♦same  research  design  utilized  for  data  collection  from  a  total  of  18  training  classes 
**dl,  d2,  d3  =  observed  training  effects 


Figure  5.  C2IPS  Training  Research  Design 

and  classes,  then  validity  of  the  design  and  the  results  are  enhanced.  In  addition,  a  formal 
study  was  selected  based  on  the  need  to  define  research  questions,  formulate  hypotheses, 
specify  data  sources,  and  develop  precise  procedures  for  collection  and  analysis  of 
research  results.  Data  were  collected  using  a  survey  method.  The  following  section 
describes  the  sample  for  this  study  in  more  detail. 

Sample 

Surveys  were  administered  at  three  separate  C2IPS  training  organizations:  the  Air 
Mobility  Warfare  Center  located  at  Fort  Dix,  New  Jersey  and  the  333rd  and  335th  Training 
Squadrons  located  at  Keesler  AFB,  Mississippi.  Copies  of  the  surveys  used  in  this  study 
are  contained  in  Appendix  E.  The  Air  Mobility  Warfare  Center  C2IPS  course  is  six 
weeks  long  and  is  designed  to  teach  students  the  responsibilities  of  air  mobility  command 
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and  control,  including  the  understanding  and  operation  of  C2IPS.  The  333rd  Training 
Squadron  provides  in-depth  C2IPS  system  administrator  training  over  a  period  of  six 
weeks,  and  gives  personnel  comprehensive  understanding  of  how  C2IPS  works  and  how 
to  maintain  the  system.  Finally,  the  335th  Training  Squadron  delivers  one-week  C2IPS 
orientation  operator  courses  to  Air  Force,  Air  National  Guard,  and  Air  Reserve  bases 
throughout  the  world.  The  335th  C2IPS  course  is  unique  in  that  instructors  travel  to 
various  Air  Force  bases  to  deliver  the  class  whereas  the  Air  Mobility  Warfare  Center  and 
333rd  courses  were  given  at  their  respective  locations.  The  335th  class  is  not 
comprehensive  but  does  provide  instruction  for  the  basic,  fundamental  operation  of 
C2IPS. 

The  sample  population  included  a  total  of  218  military,  civilian,  and  contractor 
personnel  from  throughout  the  Air  Force  who  either  operate  or  maintain  C2IPS. 

Although  most  of  the  sample  population  was  enlisted,  some  officer  and  senior  civilian 
personnel  were  also  represented  in  the  sample.  The  Air  Mobility  Warfare  Center  sample 
consisted  of  4  C2IPS  operator  classes  given  between  February  and  June  1998  for  a  total 
of  56  students.  The  333rd  sample  consisted  of  3  C2IPS  system  administrator  classes  given 
between  February  and  June  1998  for  a  total  of  15  students.  The  335th  sample  consisted  of 
12  C2IPS  operator  classes  given  between  February  and  May  1998  for  a  total  of  147 
students. 

Measures 


The  measure  of  computer  self-efficacy  used  in  this  study,  the  1 0-item  computer 
self-efficacy  scale,  was  developed  by  Compeau  and  Higgins  (1995).  As  reported  by 
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Compeau  and  Higgens,  the  computer  self-efficacy  scale  achieved  an  internal  consistency 
of  .95.  Therefore,  they  report  the  computer  self-efficacy  scale  as  highly  reliable.  The 
scale  consists  of  ten  questions,  each  reflecting  some  aspect  of  computer  self-efficacy. 

For  each  question,  respondents  are  asked  to  judge  their  ability  to  use  a  software  package 
by  answering  “yes”  or  “no”.  If  a  respondent  answers  “yes”,  they  are  then  asked  to  rate 
the  confidence  of  their  judgment  on  a  scale  of  1  to  10  with  1  meaning  “not  at  all 
confident”,  5  meaning  “moderately  confident”,  and  10  meaning  “totally  confident”.  For 
the  purposes  of  this  research,  the  scale  was  modified  slightly  to  reflect  the  type  of  training 
being  studied  (C2IPS)  and  constraints  regarding  data  collection  procedures.  The  most 
significant  change  was  eliminating  the  “yes/”no”  criterion,  and  changing  the  response 
scale  to  1  to  9  instead  of  1  to  10.  These  changes  were  made  to  simplify  reporting  and 
achieve  consistency  across  measurement  scales.  The  measure  of  computer  self-efficacy 
is  derived  by  adding  the  scores  on  each  of  the  computer  self-efficacy  items  and  dividing 
by  ten  (i.e.  the  number  of  items)  to  provide  an  overall  index  of  computer  self-efficacy. 

The  perceived  ease  of  use  scale  (Davis,  1989)  yielded  a  Cronbach  alpha  of  .93. 
The  perceived  ease  of  use  scale  consists  of  four  items.  The  respondent  is  asked  to 
indicate  the  degree  of  his/her  perceived  ease  of  use  of  a  system  on  a  7-point  Likert  scale. 
A  response  of  1  indicates  strongly  disagree,  4  indicates  neutral  (neither  agree  nor 
disagree),  7  indicates  strongly  agree.  The  scale  was  modified  to  reflect  the  use  of  C2IPS 
as  the  target  system.  The  measure  of  perceived  ease  of  use  is  derived  by  adding  the  scores 
on  each  of  the  perceived  ease  of  use  items  and  dividing  by  four  (i.e.  the  number  of  items) 
to  provide  an  overall  index  of  perceived  ease  of  use. 
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The  perceived  usefulness  scale  used  in  this  study  has  demonstrated  strong 
reliability  and  internal  consistency.  Davis  (1989)  obtained  a  Cronbach  alpha  of  .97.  The 
perceived  usefulness  scale  consists  of  four  items  indicative  of  the  construct.  The 
respondent  is  asked  to  indicate  the  degree  of  his/her  perceived  usefulness  of  a  system  on 
a  7-point  Likert  scale.  A  response  of  1  indicates  strongly  disagree,  4  indicates  neutral 
(neither  agree  nor  disagree),  and  7  indicates  strongly  agree.  As  with  computer  self- 
efficacy  and  perceived  ease  of  use,  the  perceived  usefulness  scale  was  modified  to  reflect 
the  use  of  C2IPS  as  the  target  system.  Individual  measures  of  perceived  usefulness  are 
derived  by  adding  the  scores  on  each  of  the  perceived  usefulness  items  and  dividing  by 
four  (i.e.  the  number  of  items)  to  provide  an  overall  index  of  perceived  usefulness. 

The  computer  attitude  scale  used  in  this  study  has  yielded  a  Cronbach  alpha  of  .87 
in  earlier  research  (Taylor  and  Todd,  1995:156).  The  computer  attitude  scale  consists  of 
4  items  reflecting  the  computer  attitude.  Respondents  are  asked  to  indicate  their 
computer  attitude  by  responding  to  a  7-point  Likert-type  scale.  Unlike  perceived  ease  of 
use  and  perceived  usefulness,  each  question  has  a  different  response  scale.  Examples  of 
scale  anchors  include  “Bad”  to  “Good”  and  “Dislike”  to  “Like”.  The  computer  attitude 
scale  was  modified  to  reflect  the  use  of  C2IPS  as  the  target  system.  Individual  measures 
of  computer  attitude  are  derived  by  adding  the  scores  on  each  of  the  computer  attitude 
items  and  dividing  by  four  (i.e.  the  number  of  items)  to  provide  an  overall  index  of 
computer  attitude. 

Learning  measures  were  collected  from  the  Air  Mobility  Warfare  Center.  Tests 
were  developed  by  the  training  staff  and  are  directly  related  to  course  content.  No 
reliability  measures  were  available  for  the  tests  nor  was  the  data  set  available  to  provide 
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further  analysis  of  unidimensiality.  However,  measures  of  learning  often  use  the  overall 
percentage  score  on  academic  tests.  Thus,  the  researcher  felt  that  this  measure 
represented  a  valid  measure  of  learning. 

The  performance  appraisal  instrument  used  in  this  study  was  based  on  Air 
Mobility  Command  job  performance  requirements  for  AMC  command  and  control 
personnel  and  system  administrator  personnel.  Job  performance  requirements  are  derived 
using  a  number  of  processes  and  techniques  including  formal  occupational  measurement 
by  the  Air  Education  and  Training  Command  Occupational  Measurement  Squadron  and 
subject  matter  expert  panels  (Cascio,  1991:203)  involving  Air  Force  and  major  command 
function  career  field  managers  and  training  managers.  Based  on  this  job  analysis,  AMC 
developed  a  comprehensive  list  of  behaviorally  based  performance  statements  thought  to 
include  the  scope  of  tasks  required  for  successful  C2IPS  operation  and  C2IPS  system 
administration,  for  example.  The  primary  result  of  this  process  is  the  AMC  Command 
Post/ Air  Mobility  Control  Center  Controller  Job  Performance  Requirements  List 
(AMC/DOOC,  1997:7).  This  list  provides  the  basis  for  on-the-job  training  on  C2IPS. 

The  thorough  nature  of  the  AMC  job  analysis  process  lends  more  validity  to  the  Job 
Performance  Requirements  List  as  tool  for  both  building  a  training  program  and 
evaluating  performance  (Cascio,  1991:189).  All  C2IPS  training  courses  use  at  least  some 
form  of  the  applicable  Job  Performance  Requirements  List  to  develop  course  content 
thereby  providing  an  effective  link  between  training,  practice,  and  measurement  of  on- 
the-job  performance  changes  as  a  result  of  C2IPS  training. 

The  C2IPS  Operator  performance  scale  consisted  of  24  items  related  to  C2IPS 
Operator  task  behaviors.  Each  item  was  anchored  by  a  5-point  Likert  scale  with  a 
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response  of  1  indicating  “Performs  Well  Below  Average”,  3  indicating  “Performs 
Average”,  and  5  indicating  “Performs  Well  Above  Average”  as  compared  to  their  peers. 
The  C2IPS  System  Administrator  performance  scale  consisted  of  23  items  related  to 
C2IPS  System  Administrator  task  behaviors.  A  5-point  Likert  scale  identical  to  the 
C2IPS  Operator  scale  was  used.  In  addition,  one  item  was  added  to  both  performance 
scales  asking  the  trainee  to  indicate  overall  C2IPS  performance.  This  item  also  used  the 
same  5-point  Likert  scale.  Individual  measures  of  performance  are  derived  by  adding  the 
response  magnitudes  of  each  item  on  the  performance  scale  and  dividing  by  the  total 
number  of  items  on  the  scale. 

Procedures 


Longitudinal  data  were  gathered  from  three  C2IPS  training  courses  to  track 
effects  of  training  over  time.  Surveys  of  each  training  group  measured  trainee  computer 
self-efficacy,  perceived  ease  of  use,  perceived  usefulness,  computer  attitude,  and  self- 
reported  performance  at  the  beginning  of  the  training  course,  at  the  end  of  the  training 
course,  and  on  the  job.  In  addition,  a  measure  of  trainee  on-the-job  performance  using 
C2IPS  was  provided  by  each  trainee’s  supervisor.  For  all  studies,  the  “Pre-Then-Post” 
technique  (see  Chapter  II)  for  reducing  self-report  bias  was  employed.  This  technique, 
coupled  with  behaviorally-specific  performance  statements  and  reliable  MIS 
measurement  instruments  was  used  to  increase  the  quality  of  self-report  performance 
data. 

All  trainees  completed  self-report  measures  of  computer  self-efficacy,  perceived 
ease  of  use,  perceived  usefulness,  computer  attitude,  and  self-rated  performance  using 
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C2IPS  at  the  beginning  of  the  class  (time  1),  end  of  the  class  (time  2),  and  on-the-job 
(time  3).  In  addition,  trainee  supervisors  completed  a  post-training  survey  of  on-the-job 
performance  for  the  trainee.  Surveys  were  administered  by  C2IPS  training  personnel 
during  the  course,  and  on-the-job  surveys  were  mailed  to  the  trainee  and  the  trainee’s 
supervisor.  Academic  tests  were  administered  as  a  measure  of  learning  by  the  Air 
Mobility  Warfare  Center.  The  Air  Mobility  Warfare  Center  utilized  pre-  and  post-tests. 

Finally,  a  variety  of  demographic  data  were  collected  including  Rank,  Grade, 
Enlisted  Skill  Level,  Age,  Gender,  Type  of  Organization,  Self-Reported  Usage  of  C2IPS, 
and  Previous  Experience  on  C2IPS.  Although  not  explicitly  part  of  the  C2IPS  training 
model,  these  data  were  collected  with  regard  to  the  relationship  they  might  have  with 
increases  in  performance  and  to  provide  a  description  of  the  sample.  Type  of 
Organization  indicated  whether  the  trainee  works  in  a  base  level,  numbered  air  force, 
MAJCOM,  or  other  organization.  Self-Reported  Usage  of  C2IPS  was  a  measure  of  how 
much  time  the  trainee  perceived  he/she  spent  on  C2IPS  on  any  given  day.  Self-Reported 
Usage  was  measured  in  hours  using  a  6-point  scale  where,  for  example,  a  response  of  1 
meant  “no  time  spent  on  C2IPS”,  3  meant  “1-3  hours  spent  on  C2IPS  per  day,”  etc. 
Previous  Experience  on  C2IPS  reported  the  amount  of  months  the  trainee  had  worked 
with  C2IPS  before  attending  the  training  course 

Pilot  Study 

A  pilot  study  of  survey  instrumentation  was  conducted  using  the  responses  of  20 
Air  Force  Institute  of  Technology  graduate  students.  Based  on  the  feedback  from  the 
pilot  study,  several  changes  to  the  survey  were  made  to  facilitate  readability, 
comprehension,  and  speed  of  completion.  Modified  versions  of  the  surveys  were  sent  to 
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the  Air  Force  Personnel  Center’s  Survey  Control  Office.  After  minor  wording  changes, 
the  surveys  were  approved  for  this  study  and  assigned  a  Survey  Control  Number  of 
USAF  SCN  98-33. 

Statistical  Analysis  and  Guidelines  for  Evaluating  Hypothesis 

Statistical  Analysis.  The  data  collected  from  C2IPS  training  course  surveys  are 
analyzed  using  a  variety  of  statistical  techniques  including  descriptive  statistics, 
reliability  analysis,  factor  analysis,  normality  tests,  t-tests,  and  multiple  regression. 
Descriptive  statistics  such  as  means  and  standard  deviations  are  used  to  prepare  the  data 
for  further  analysis  and  provide  a  general  idea  of  the  relationships  between  variables. 
Test-retest  reliability  analysis  using  Cronbach’s  alpha  is  performed  to  determine  the 
stability  of  the  computer  self-efficacy,  perceived  ease  of  use,  perceived  usefulness, 
computer  attitude,  and  performance  instruments  when  given  to  the  trainees  at  each  stage 
of  the  study  to  assess  the  impact  of  changes  made  to  the  scales.  Factor  analysis  is  used  to 
verify  the  overall  construct  validity  and  acceptability  of  the  survey  items. 

Evaluation  of  Propositions  and  Analysis  of  Hypotheses.  Although  the  above 
analyses  are  straightforward,  the  variety  and  number  of  training  classes  as  well  as  the 
longitudinal  nature  of  this  study  makes  evaluating  hypotheses  in  terms  of  statistical 
results  more  complex.  For  instance,  if  a  hypothesis  using  a  variable  is  significant  at  the 
end  of  training  but  not  on-the-job,  is  the  hypothesis  supported  or  not?  Despite  this 
quandary,  suitable  evaluation  guidelines  can  be  developed  using  statements  from  the 
discussion  of  training  presented  in  Chapter  II  that  will  help  classify  whether  or  not  a 
hypothesis  is  supported.  Chapter  II  stated  that  the  focus  of  the  Air  Force  evaluation  is  to 
ensure  that  training  programs  ensure  that  graduates  acquire  the  skills,  knowledge,  and 
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attitudes  to  do  the  job;  that  these  graduates  meet  job  performance  requirements;  and  to 
identify  ways  to  improve  the  graduate’s  performance  (AFI 36-2201 :7).  In  addition, 
Wexley  and  Latham  (1991)  and  Goldstein  (1974)  state  that  transfer  of  training  is  crucial. 
Using  these  statements  as  a  framework,  three  basic  guidelines  can  be  developed. 

Guideline  1 :  increases  in  performance,  learning,  and  attitudes  as  a  result  of 
training  are  desirable. 

Guideline  2:  the  transfer  to  the  job  of  performance,  learning,  and  attitudes  that 
were  higher  than  trainee  had  before  training  is  desirable.  In  terms  of  relative  importance, 
guideline  2  is  more  important  than  guideline  1 . 

Guideline  3:  factors  that  influence  trainee  performance  during  training  and  on  the 
job  are  desirable. 

Hypotheses  related  to  proposition  1 ,  whether  or  not  training  increases  attitudes, 
learning,  and  performance,  will  be  evaluated  using  paired  t-test.  Under  the  terms  of 
guidelines  1  and  2,  these  hypotheses  will  be  fully  supported  if  they  show  a  transfer  effect. 
In  statistical  terms,  this  means  that  paired  t-tests  will  be  significant  either  from  time  1 
(beginning  of  training)  to  time  3  (on-the-job)  or  from  time  2  (end  of  training)  to  time  3. 
Hypotheses  related  to  proposition  1  will  be  partially  supported  if  they  show  only  a 
training  effect.  Statistically,  this  means  that  paired  t-tests  will  be  significant  only  from 
time  1  to  time  2. 

Hypotheses  related  to  propositions  2  and  3  will  be  analyzed  using  stepwise  linear 
regression.  These  propositions  ask  whether  computer  self-efficacy,  perceived  usefulness, 
perceived  ease  of  use,  and  computer  attitude  influence  learning  and  performance. 
Guideline  3  is  therefore  appropriate  for  evaluating  these  hypotheses.  In  terms  of 
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guideline  3,  this  means  that  a  variable  that  predicts  performance  as  part  of  a  linear 
regression  model  is  desirable.  In  statistical  terms,  hypotheses  related  to  propositions  2 
and  3  will  be  fully  supported  if  the  regression  model  is  significant  (F-value)  and  the 
variable  in  the  regression  model  is  significant  (t-value). 

Longitudinal  Assessment  of  Variables  Using  Linear  Regression.  One  of  the 
benefits  of  a  longitudinal  study  is  that  changes  in  variables  over  time  may  be  assessed. 
Multiple  paired  t-tests  like  those  described  above  provide  such  an  opportunity,  but  the 
same  idea  using  linear  regression  much  be  approached  a  bit  more  systematically. 
Specifically,  the  impact  of  independent  variables  as  measured  at  a  previous  time  in  the 
study  (time  1 ,  for  instance)  to  predict  the  dependent  variable  as  measured  at  a  later  time 
(time  3)  would  give  an  idea  of  the  robustness  of  the  independent  variable  in  predicting 
the  dependent  variable.  In  other  words,  if  the  independent  variable  remains  significant 
(provided  the  regression  models  are  significant)  over  time,  then  it  may  be  viewed  as  a 
reliable  predictor  of  the  dependent  variable.  Chapter  IV  implements  this  idea  using  3 
linear  regression  equations  with  learning  as  the  dependent  variable  and  6  linear  regression 
equations  with  self-rated  performance  as  the  dependent  variable.  Table  1  shows  the  list 
of  equations. 
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Table  1.  Linear  Regression  Equations 


Time  1  Learning  =Time  1  Computer  Self-Efficacy+Time  1  Perceived  Usefulness 

+Time  1  Perceived  Ease  ofUse+Time  1  Computer  Attitude+Usage 

Time  2  Leaming=Time  1  Computer  Self-Efficacy+Time  1  Perceived  Usefulness 

+Time  1  Perceived  Ease  ofUse+Time  1  Computer  Attitude+Usage 

Time  2  Learning^Time  2  Computer  Self-Efficacy+Time  2  Perceived  Usefulness 

+Time  2  Perceived  Ease  ofUse+Time  2  Computer  Attitude+Usage 

Time  1  Performance=Time  1  Computer  Self-Efficacy  +T»me  1  Perceived  Usefulness 

+Time  1  Perceived  Ease  ofUse+Time  1  Computer  Attitude+Usage 

Time  2  Performance=Time  1  Computer  Self-Efficacy+Time  1  Perceived  Usefulness 

+Time  1  Perceived  Ease  ofUse+Time  1  Computer  Attitude+Usage 

Time  2  Performance=Time  2  Computer  Self-Efficacy+Time  2  Perceived  Usefulness 

+Time  2  Perceived  Ease  ofUse+Time  2  Computer  Attitude+Usage 

Time  3  Performance=Time  1  Computer  Self-Efficacy+Time  1  Perceived  Usefulness 

+Time  1  Perceived  Ease  ofUse+Time  1  Computer  Attitude+Usage 

Time  3  Performance=Time  2  Computer  Self-Efficacy+Time  2  Perceived  Usefulness 

+Time  2  Perceived  Ease  ofUse+Time  2  Computer  Attitude+Usage 

Time  3  Performance=Time  3  Computer  Self-Efficacy  +Time  3  Perceived  Usefulness 

+Time  3  Perceived  Ease  ofUse+Time  3  Computer  Attitude+Usage 
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IV.  Results 


Chapter  Overview 

Chapter  IV  presents  the  analysis  and  results  of  the  C2IPS  training  surveys 
administered  to  the  335th  Training  Squadron,  Air  Mobility  Warfare  Center,  and  333rd 
Training  Squadron.  First,  the  sample  size,  demographics,  and  distribution  of  responses 
will  be  discussed.  Next,  the  reliability  analysis  of  the  survey  scales  will  be  presented. 

The  hypotheses  related  to  Proposition  1,  involving  the  relationship  between  training  and 
trainee  reactions,  are  discussed  using  the  results  of  paired  t-tests  of  key  variables.  In 
addition,  the  means  of  on-the-job  self-rated  and  supervisor-rated  performance  are 
compared  using  an  independent  t-test.  Then,  the  outcomes  of  hypothesis  related  to 
Proposition  2,  the  relationship  between  trainee  reactions  and  trainee  learning,  is  described 
using  the  results  of  linear  regression  analyses.  Finally,  hypotheses  related  to  Proposition 
3,  the  relationship  of  trainee  reactions  and  trainee  performance,  is  similarly  presented  in 
terms  of  the  results  of  linear  regression  analyses.  Conclusions  based  on  these  results  will 
be  presented  in  Chapter  V. 

Survey  Results 

General  Demographics.  C2IPS  training  surveys  (Appendix  B)  were  administered 
to  207  military,  civilian,  and  contractor  personnel.  As  seen  in  Table  2,  the  335th  Training 
Squadron  had  by  far  the  most  participants  while  the  333rd  Training  Squadron  has  the 
smallest  number  of  participants.  All  surveys  were  reviewed  for  completeness  and 
usability.  Of  particular  note  is  the  fact  that  data  from  10  trainee  surveys  from  the  Air 
Mobility  Warfare  Center  survey  were  unusable  due  to  faulty  survey  completion.  These 
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10  trainees  used  the  beginning  of  course  survey  instead  of  the  end  of  course  survey  to 
register  their  responses  on  the  end  of  their  training  class.  To  collect  on-the-job  measures 
(time  3),  surveys  were  sent  to  1 69  of  the  207  trainees  and  their  supervisors.  Surveys  were 
not  sent  to  all  trainees  and  supervisors  due  to  the  constraints  placed  on  timely  data 
collection  and  the  completion  of  this  report.  A  total  of  52  usable  trainee  surveys  were 
returned  for  a  response  rate  of  3 1  %,  and  a  total  of  26  supervisor  surveys  were  returned 
for  a  response  rate  of  15.5%.  Additional  histograms  and  normality  plots  of  selected 
variables  are  contained  in  Appendix  A.  All  calculations  were  performed  using  SPSS® 
Graduate  Pack  8.0  for  Windows. 


Table  2.  Sample  Size  by  Course  and  Class 


335th  Air  Mobility  Warfare  Center  333rd 


Class  1 

15 

Class  1 

18 

Class  1 

10 

Class  2 

15 

Class  2 

15 

Class  2 

5 

Class  3 

10 

Class  3 

3 

TOTAL 

15 

Class  4 

7 

Class  4 

9 

Class  5 

12 

TOTAL 

45 

Class  6 

13 

Class  7  9 

Class  8  6 

Class  9  13 

Class  10  19 

Class  11  10 

Class  12  18 

TOTAL  147 

OVERALL  SAMPLE  SIZE  =  207 


To  build  a  picture  of  the  overall  survey  sample,  Tables  3  through  9  provide  frequency 
statistics  from  the  study’s  demographic  variables.  It  should  be  noted  that  although  the 
overall  sample  size  was  207,  sample  sizes  for  many  of  the  demographic  variables  do  not 
equal  207.  This  can  be  attributed  to  omissions  on  the  survey  response  sheets,  obvious 
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errors  (e.g.  out-of-range  responses),  and  indeterminate  responses  (e.g.  multiple  responses 
for  the  same  item). 

As  shown  in  Table  3,  enlisted  personnel  comprised  the  overwhelming  majority  of 
the  sample  at  72%.  Officers,  civilians,  and  contractor  personnel  were  evenly  distributed 
at  about  8%  each.  This  distribution  is  expected  because  most  C2IPS  operators  and 
system  administrators  are  enlisted  personnel. 

Table  3.  Frequencies  Based  on  Rank 

RANK 


Category  Frequency  Percent 
Enlisted  150  72 

Officer  17  8 

Civilian  (WG)  3  1 

Civilian  (GS)  15  7 

Contractor  17  8 

Other  2  1 

Total  204 


Table  4  shows  that,  for  enlisted  personnel,  all  skill  levels  were  relatively  evenly 
represented.  However,  based  on  the  proportion  of  enlisted  personnel  in  Table  3,  there  is 
some  degree  of  error  in  Table  3.  In  Table  3,  the  total  number  of  enlisted  is  150. 
Therefore,  3-,  5-,  and  7-level  frequencies  should  total  150.  As  seen  in  Table  4,  they  do 
not.  As  such,  this  information  should  be  viewed  with  some  skepticism. 

Most  respondents  were  between  the  ages  of  20  and  30  (52%)  followed  by  ages  31 
to  40  (27%).  This  ratio  is  generally  consistent  with  Air  Force  demographic  statistics  for 
age  ranges  of  its  personnel  (Air  Force  Personnel  Center,  1998). 
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Table  4.  Frequencies  Based  on  Enlisted  Skill  Level 


ENLISTED  SKILL  LEVEL 


Category 

Frequency 

Percent 

3 -Level 

44 

21 

5-Level 

60 

29 

7-Level 

59 

28 

N/A 

40 

19 

Total 

203 

Table  5. 

Frequencies 

Base< 

AGE 

Category 

Frequency 

Percent 

Under  20 

14 

7 

20  to  30 

107 

52 

31  to  40 

55 

27 

41  to  50 

20 

10 

51  to  60 

7 

3 

Over  60 

1 

0 

Total 

204 

As  with  age,  gender  is  again  consistent  with  Air  Force  gender  ratios  of 
approximately  82%  male  to  18%  female  (Air  Force  Personnel  Center,  1998). 


Table  6.  Frequencies  Based  on  Gender 


GENDER 

Category 

Frequency 

Percent 

Male 

152 

72 

Female 

49 

23 

Total 

201 
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In  addition,  Table  7  shows  that  Wing  and  Base  level  C2IPS  personnel  comprised 
the  overwhelming  majority  of  participants.  “Other”  organizations  included  contractor 
companies  and  Air  Force  Reserve  units.  Numbered  Air  Force  and  Major  Command 
respondents  rounded  out  the  rest  of  the  sample.  C2IPS  systems  are  predominately  fielded 
at  Wing  and  Base  level  organizations  with  a  much  smaller  proportion  fielded  at  the 
Numbered  Air  Force  and  Major  Command  level.  Therefore,  one  would  expect  the 
majority  of  C2IPS  personnel  to  work  in  Wing  and  Base  level  organizations. 

Table  7.  Frequencies  Based  on  Organization 


ORGANIZATION 

Category 

Frequency 

Percent 

Wing/Base 

139 

67 

NAF 

9 

4 

MAJCOM 

17 

8 

Other 

36 

17 

Total 

201 

Also,  it  is  not  surprising  that  most  trainees  had  little  or  no  previous  experience  with 
C2IPS  (76%).  This  is  to  be  expected  from  a  training  course  (see  Table  8). 


Table  8.  Frequencies  Based  on  Previous  C2IPS  Experience 


PREVIOUS  C2IPS  EXPERIENCE 


Category 

Frequency 

Percent 

None 

97 

47 

1  to  6  Months 

61 

29 

7  to  12  Months 

18 

9 

>  12  Months 

23 

11 

Total 

199 
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Similarly,  Table  9  demonstrates  that  most  trainees  perceived  that  they  used 
C2IPS  less  than  3  hours  during  the  duty  day  (74%).  For  these  personnel,  C2IPS  may 
serve  as  only  a  part  of  their  overall  duties.  In  contrast,  the  remaining  22%  of  trainees 
perceived  they  used  C2IPS  for  more  than  4  hours  per  day.  For  these  personnel,  C2IPS 
may  be  viewed  as  the  predominant  function  of  their  job. 


Table  9.  Frequencies  Based  on  Self-reported  usage  of  C2IPS 


SELF-REPORTED  USAGE  OF  C2IPS 


Category  Frequency  Percent 


None 

76 

37 

<  1  hour 

40 

19 

1  to  3  hours 

37 

18 

4  to  6  hours 

24 

12 

7  to  8  hours 

20 

10 

>  8  hours 

4 

2 

Total 

201 

Scale  Reliabilities.  Calculations  on  all  variable  scales  produced  acceptable  levels 
of  internal  consistency,  as  shown  in  Table  10.  In  addition,  Cronbach  Alphas  were 
generally  consistent  with  values  obtained  from  previous  research,  as  described  in  Chapter 
II.  Based  on  the  Cronbach  Alphas,  items  measuring  specific  constructs  can  be  assumed 
to  have  high  internal  consistency  and  are  able  to  produce  consistent  results  over  time  with 
successive  administrations  (Cooper  and  Emery,  1995:153).  The  C2IPS  performance 
scales  were  particularly  noteworthy  for  their  high  reliabilities  (a  =  .96-. 99).  In  addition, 
a  factor  analysis  using  all  items  from  the  computer  self-efficacy,  perceived  usefulness, 
perceived  ease  of  use,  computer  attitude,  and  self-rated  performance  scales  was 
performed  (results  are  contained  in  Appendix  B).  The  factor  analysis  showed  that  all 
variable  items  appeared  to  factor  into  distinct  categories  representing  the  constructs 
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purported  to  be  captured  by  the  variable  scales.  However,  the  results  should  be  viewed 
with  caution  due  to  the  small  sample  size  to  item  ratio.  Experts  generally  agree  that  the 
ratio  for  a  reliable  factor  analysis  should  be  around  10:1.  The  ratio  for  this  study  was  207 
respondents  to  45  items  or  about  4.6:1. 

Table  10.  Scale  Reliabilities,  All  Courses:  Time  1  and  Time  2 

VARIABLES: 


CRONBACH  ALPHAS  BY  COURSE 

Time  1 

Time  2 

Time  3 

Computer  Self-Efficacy: 

Overall  (All  Courses) 

.94 

.93 

.94 

335,h 

.93 

.94 

X 

AMWC 

.94 

.92 

X 

333rd 

.96 

.96 

X 

Perceived  Usefulness: 

Overall  (All  Courses) 

.92 

.94 

.96 

335lh 

.91 

.94 

X 

AMWC 

.94 

.93 

X 

333rd 

.84 

.96 

X 

Perceived  Ease  of  Use: 

Overall  (All  Courses) 

.85 

.81 

.86 

335“' 

.84 

.85 

X 

AMWC 

.86 

.65 

X 

333*“ 

.79 

.90 

X 

Computer  Attitude: 

Overall  (All  Courses) 

.92 

.90 

.89 

335“' 

.92 

.92 

X 

AMWC 

.92 

.83 

X 

333rd 

.92 

.94 

X 

Self-Report  Performance: 

Overall  (All  Courses) 

.99 

.97 

.97 

335'" 

.99 

.96 

X 

AMWC 

.99 

.97 

X 

333rd 

.99 

.97 

X 

Rater  On-the-Job  Performance  Eval  (Time  3  Only): 

Overall  (All  Courses) 

X 

X 

.9823 

Proposition  1 .  Information  technology  training  increases  a  trainee’s  reactions, 
learning,  and  performance. 

Paired  t-tests  were  performed  for  Computer  Self-Efficacy,  Perceived  Usefulness, 
Perceived  Ease  of  Use,  Computer  Attitude,  Learning,  and  Performance  measures  to 
determine  the  relationship  of  training  on  these  variables  over  time  and  to  determine  the 
supportability  of  Proposition  1  and  its  related  hypothesis.  Comparisons  were  made 
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between  Time  1  (Beginning  of  Course)  and  Time  2  (End  of  Course),  Time  2  and  Time  3 
(On-the-Job),  and  Time  1  and  Time  3.  Hypotheses  related  to  Proposition  1  are  evaluated 
in  terms  of  these  comparisons. 

As  a  precautionary  measure,  an  independent  t-test  was  conducted  between  Time  3 
self-rated  performance  and  supervisor-rated  performance  to  determine  if  any  bias  existed 
in  either  of  the  ratings.  Based  on  the  literature  examined  in  Chapter  II,  subordinates  have 
been  reported  to  systematically  inflate  their  performance  ratings  as  compared  to  those  of 
their  supervisors.  Therefore,  if  the  t-test  detects  no  significant  difference,  then  self-rated 
performance  indexes  can  be  assumed  to  be  a  fairly  reliable  measure  of  C2IPS 
performance  following  training  and,  by  extension,  during  training. 

Table  1 1  shows  t-tests  comparing  Time  1  and  Time  2  scores  for  each  C2IPS 
course.  Based  on  the  number  of  participants  in  each  course,  the  variables  measured  in 
each  course  were  either  significant  or  showed  a  trend  toward  increased  significance.  For 
example,  although  all  333rd  relationships  were  not  significant,  the  magnitudes  of 
significance  are  likely  attributed  to  the  low  number  of  participants  (N  =  15).  It  is 
reasonable  to  assume  that  a  higher  N  would  yield  comparable  levels  of  significance. 
Therefore,  aside  from  the  results  presented  in  Table  10,  all  future  analyses  will  utilize  a 
combined  measure  of  all  cases  from  the  three  training  courses.  More  detailed  discussions 
of  hypotheses  related  to  Proposition  1  follow. 

Hypothesis  1:  There  will  be  a  significant  positive  increase  in  a  trainee ’s 
computer  self-efficacy  as  a  result  of  C2IPS  training. 

The  results  of  the  overall  paired  t-test  from  time  1  to  time  2  (Table  12)  show  an 
increase  in  computer  self-efficacy  from  an  average  of  50.85  at  time  1  to  66.01  at  time  2 
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(t  =  - 1 2. 1 9,  p  <  .00 1 ).  However,  no  significant  change  was  detected  between  time  2  and 
time  3  meaning  computer  self-efficacy  stayed  at  the  same  level  when  the  trainee  returned 
to  the  job  (Table  13).  Although  no  increase  was  detected  after  the  end  of  training, 
computer  self-efficacy  still  showed  a  significant  increase  from  time  1  to  time  3  from 
50.00  to  65.10  (t  =  -6.29,  p  <  .001)  as  seen  in  Table  14.  Therefore,  Hypothesis  1  is 
therefore  supported. 

Hypothesis  4:  There  will  be  a  significant  positive  increase  in  a  trainee ’s 
perceived  ease  of  use  of  C2IPS  following  CUPS  training. 

As  Table  12  shows,  perceived  ease  of  use  increased  from  a  score  of  17.35  at  time 
1  to  20.15  at  time  2  (t  =  -1 1 .54,  p  <  .001).  Like  computer  self-efficacy,  no  significant 
change  in  the  level  of  perceived  ease  of  use  was  noted  between  time  2  and  time  3  (Table 
13).  This  means  perceived  ease  of  use  remained  the  same  after  the  trainee  resumed 
his/her  normal  duties.  However,  Table  14  shows  that  the  change  from  time  1  to  time  3 
remained  significant  (t  =  -5.04,  p  <  .001).  As  a  result,  Hypothesis  4  is  supported. 

Hypothesis  7:  There  will  be  a  significant  positive  increase  in  a  trainee ’s 
perceived  usefulness  of  C2IPS  as  a  result  of  C2IPS  training. 

Perceived  usefulness  of  C2IPS  was  higher  at  time  2  (22.40)  than  at  time  1  (20.12). 
In  addition,  the  relationship  was  significant  at  the  .001  level  (Table  12).  However,  from 
time  2  to  time  3,  dropped  from  22.40  to  20.98  (t=  2.14,  p  <  .05).  This  drop  resulted  in  a 
score  that  was  similar  to  the  score  obtained  from  the  trainee  at  the  beginning  of  the 
training  course:  20.92  at  time  1  vs.  20.98  at  time  3.  So,  although  training  temporarily 
boosted  perceived  usefulness  immediately  following  training,  the  increase  was  not 
sustained  when  the  trainee  returned  to  the  job.  Therefore,  Hypothesis  7  was  partially 
supported. 
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Table  11.  Paired  t-tests,  Time  1  and  Time  2:  Grouped  by  Variable  and 

Course 


Overall  (All  Courses) 

Time  1  Time  2 


Variable 

Mean 

Std.  Dev 

Mean 

Std.  Dev 

df 

t 

Computer  Self-Efficacy 

50.85 

18.33 

66.01 

16.03 

202 

-12.19*** 

Perceived  Usefulness 

20.12 

5.84 

22.40 

5.71 

203 

-6.092*** 

Perceived  Ease  of  Use 

17.35 

5.49 

20.15 

6.08 

203 

-11.536*** 

Computer  Attitude 

19.93 

5.30 

22.10 

4.95 

202 

-6.712*** 

Self-Report  Performance 

2.67 

1.17 

3.92 

0.70 

200 

-18.375*** 

335th 

Time  1 

Time  2 

Variable 

Mean 

Std.  Dev 

Mean 

Std.  Dev 

df 

t 

Computer  Self-Efficacy 

52.75 

17.86 

67.14 

16.29 

143 

-9.88*** 

Perceived  Usefulness 

20.37 

5.70 

22.45 

5.57 

144 

-3.03*** 

Perceived  Ease  of  Use 

17.84 

5.39 

21.01 

5.31 

144 

-8.33*** 

Computer  Attitude 

20.32 

5.26 

22.36 

5.00 

143 

-5.27*** 

Self-Report  Performance 

2.74 

1.17 

3.96 

0.72 

142 

-15.50*** 

Air  Mobility  Warfare  Center 


Time  1  Time  2 


Variable 

Mean 

Std.  Dev 

Mean 

Std.  Dev 

df 

t 

Computer  Self-Efficacy 

45 

18.9413 

61.98 

15.63611 

43 

-6.01*** 

Perceived  Usefulness 

19.64 

6.7795 

22.3 

6.245294 

43 

-3.63*** 

Perceived  Ease  of  Use 

16.8 

5.719186 

17.48 

7.656852 

43 

-0.58*** 

Computer  Attitude 

19.24 

5.89744 

21.89 

4.95663 

43 

-3.73*** 

Self-Report  Performance 

2.423 

1.180927 

3.807 

0.657898 

42 

-8.73*** 

Learning 

32.61 

17.56714 

86.28 

7.937654 

17 

-11.27*** 

333rd 

Time  I 

Time  2 

Variable 

Mean 

Std.  Dev 

Mean 

Std.  Dev 

df 

t 

Computer  Self-Efficacy 

49.93 

18.37 

67.00 

13.34 

14 

-3.76** 

Perceived  Usefulness 

19.13 

4.00 

21.53 

5.18 

14 

-1.38 

Perceived  Ease  of  Use 

14.13 

4.79 

19.27 

5.15 

14 

-3.06** 

Computer  Attitude 

18.13 

2.77 

20.20 

4.28 

14 

-1.92* 

Self-Report  Performance 

2.78 

1.08 

3.82 

0.50 

14 

-4.58** 

*  Significant  at  p  <  .  1 

**  Significant  at  p  <  .01 

***  Significant  at  p  <  .001 
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Hypothesis  10:  There  will  be  a  significant  positive  increase  in  a  trainee ’s 
computer  attitude  following  C2IPS  training. 

Table  12  shows  that  computer  attitude  increased  from  time  1  to  time  2  (t  =  -6.712, 
p  <  .001).  Like  perceived  usefulness,  however,  this  increase  did  not  transfer  to  the  job 
and  in  fact  fell  to  the  same  level  reported  by  the  trainee  at  the  beginning  of  the  training 
course:  20.56  at  time  1  vs.  20.98  at  time  3  (t-test  non-significant)  as  seen  in  Table  14.  As 
a  result,  Hypothesis  1 0  was  partially  supported. 

Figure  7  summarizes  the  paired  t-tests  for  each  variable  in  graphical  format.  The 
lines  represent  the  percentage  increase  or  decrease  in  each  variable  over  time.  From  the 
Figure,  the  dramatic  increases  in  self-rated  performance,  computer  self-efficacy,  and 
perceived  ease  of  use  are  clearly  shown.  In  addition,  the  rise  and  fall  of  perceived 
usefulness  and  computer  attitude  over  time  is  easily  seen. 

Table  12.  Overall  Paired  t-tests,  Time  1  and  Time  2:  Grouped  by  Variable 


Time  1 


Variable 

Mean 

Std.  Dev 

Computer  Self-Efficacy 

50.85 

18.33 

Perceived  Usefulness 

20.12 

5.84 

Perceived  Ease  of  Use 

17.35 

5.49 

Computer  Attitude 

19.93 

5.30 

Self-Report  Performance 

2.67 

1.17 

Significant  at  p  <  .001 


Overall  (All  Courses) 


Time  2 


Mean 

Std.  Dev 

df 

t 

66.01 

16.03 

202 

-12.19*** 

22.40 

5.71 

203 

-6.092*** 

20.15 

6.08 

203 

-11.536*** 

22.10 

4.95 

202 

-6.712*** 

3.92 

0.70 

200 

-18.375*** 
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Table  13.  Overall  Paired  t-tests,  Time  2  and  Time  3:  Grouped  by  Variable 


Overall  (All  Courses) 


Time  2 


Variable 

Mean 

Std.  Dev 

Computer  Self-Efficacy 

63.51 

18.04 

Perceived  Usefulness 

22.71 

5.90 

Perceived  Ease  of  Use 

21.82 

4.38 

Computer  Attitude 

22.98 

4.13 

Self-Report  Performance 

3.97 

.6626 

Time  3 


Mean 

Std.  Dev 

df 

t 

65.49 

15.52 

48 

-.679 

20.98 

6.80 

48 

2.144* 

21.77 

5.20 

48 

.055 

20.92 

4.82 

48 

2.919** 

3.68 

.88 

48 

2.501* 

Significant  at  p  <  .05 
Significant  at  p  <  .001 


Table  14.  Overall  Paired  t-tests,  Time  1  and  Time  3:  Grouped  by  Variable 


Time  1 


Variable 

Mean 

Std.  Dev 

Computer  Self-Efficacy 

50.00 

19.40 

Perceived  Usefulness 

20.92 

6.55 

Perceived  Ease  of  Use 

18.1 

5.56 

Computer  Attitude 

20.56 

5.28 

Self-Report  Performance 

2.74 

1.30 

*  Significant  at  p<. 001 


Overall  (All  Courses) 


Time  3 


Mean 

Std.  Dev 

df 

t 

65.10 

15.60 

49 

-6.29* 

20.98 

6.74 

49 

-.081 

21.82 

5.16 

49 

-5.04* 

20.98 

4.79 

49 

-.68 

3.68 

.88 

45 

-6.07* 

Time  1  Time  2  Time  3 


Computer  Self- 
Efficacy 


Perceived 

Usefulness 


Perceived 
Ease  of  Use 


Computer 

Attitude 


—•—Self-Rated 

Performance 


Figure  6.  Percent  Change  in  Reactions  and  Performance  Over  Time 
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Performance  and  Learning.  Next,  increases  in  performance  and  learning  were 


analyzed  using  paired  t-tests.  Although  increases  in  these  relationships  were  not 
explicitly  hypothesized  for  this  study,  it  is  useful  to  analyze  the  data  if  only  to  lend 
credibility  to  the  C2IPS  training  effort  and  this  research. 

As  seen  in  Table  12,  self-rated  performance  on  C2IPS  showed  a  dramatic  increase 
from  time  1  to  time  2  of  (t  =  -1 8.38,  p  <  .001).  From  time  2  to  time  3,  self-rated 
performance  dropped  from  3.92  to  3.68  (t  =  2.50,  p  <  .05)  between  the  end  of  the  training 
course  and  on-the-job.  However,  when  compared  with  beginning  of  course  performance 
levels  and  on  the  job  performance  levels,  self-rated  performance  increased  significantly 
from  2.74  to  3.68  (t  =  -6.07,  p  <  .001). 

Learning  also  showed  a  dramatic  increase  over  time.  As  Table  1 1  shows,  trainees 
scored  an  average  of  32.61%  on  the  beginning  of  the  course  academic  test  compared  with 
an  average  of  86.28%  at  the  end  of  the  course  test  (t  -  -1 1 .27,  p  <  .001). 

In  addition,  an  independent  t-test  of  time  3  on-the-job  self-rated  performance 
measures  and  on-the-job  supervisor-rated  performance  measures  was  performed.  This 
test  was  conducted  to  show  if  there  were  differences  in  the  way  trainees  and  supervisors 
perceived  performance  on  C2IPS  following  training.  Previous  research  has  indicated  a 
systematic  inflationary  bias  when  subordinates  rate  their  performance  as  compared  to 
their  supervisors.  An  independent  t-test  yielding  a  significant  difference  would  mean  that 
bias  occurred  while  a  non-significant  difference  would  mean  that  trainees  and  supervisors 
rated  performance  on  C2IPS  in  a  similar  manner. 

As  shown  in  Table  15,  trainee  and  supervisor  C2IPS  performance  ratings  showed 
no  significant  difference  (t  =  -.01,  p  =  .99).  In  fact,  trainees  and  supervisors  rated  C2IPS 
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performance  nearly  identically.  Based  on  this  result,  trainee  self-report  C2IPS 
performance  measures  are  assumed  to  be  credible  measures  and  valid  surrogates  of 
C2IPS  performance. 

Table  15.  Independent  t-test,  Time  3:  Comparison  of  On-the-Job  Self-Rated 
Performance  and  On-the-Job  Supervisor-Rated  Performance 


On-the-Job  Supervisor  Rated  Performance  On-the-Job  Self-Rated  Performance 


Mean 

3.63 

3.63 

Variance 

0.94 

0.86 

Observations 

26 

49 

Pooled  Variance 

0.89 

Hypothesized  Mean  Difference 

0 

df 

73 

t 

-0.01 

P(T<=t),  two-tail 

0.99 

t  Critical,  two-tail 

1.99 

Proposition  2.  The  higher  a  trainee’s  reactions  to  IT  training,  the  higher  the 
trainee’s  learning. 

Hypothesis  2:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  computer  self-efficacy  and  trainee  learning  on  C2IPS. 

Hypothesis  5:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  perceived  ease  of  use  of  C2IPS  and  trainee  learning  on  C2IPS. 

Hypothesis  8:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  perceived  usefulness  of  C2IPS  and  trainee  learning. 

Hypothesis  11:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  computer  attitude  and  trainee  learning. 

The  Air  Mobility  Warfare  Center  provided  the  only  opportunity  to  examine  this 
proposition  and  its  associated  hypotheses,  as  it  was  the  only  course  to  offer  beginning  and 
end  of  course  academic  knowledge  testing.  In  addition,  because  pre-  and  post-  testing  of 
trainees  was  just  getting  underway,  disparities  in  test  administration  yield  a  total  of  1 8 


63 


cases  at  time  1  and  34  cases  at  time  2.  Three  multiple  regression  analyses  were 
performed  using  results  of  academic  knowledge  test  scores  (learning)  as  the  dependent 
variable  and  four  independent  variables  contained  in  the  hypotheses.  Therefore, 
proposition  2  and  its  hypotheses  were  not  supported. 

(NOTE:  As  Table  16  shows,  none  of  the  three  regression  models  were  significant  at  an 
a  =1  level.  Since  none  of  the  models  and  variables  were  significant  at  the  a  =.l  level, 
p-values  for  the  models  and  variables  are  not  shown  in  Table  16.) 

Proposition  3.  The  higher  a  trainee’s  reactions  to  IT  training,  the  better  the 
trainee’s  on-the-job  performance. 

Hypothesis  3:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  computer  self-efficacy  and  trainee  performance  on  C2IPS. 

Hypothesis  6:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  perceived  ease  of  use  of  C2IPS  and  trainee  performance  on  C2IPS. 

Hypothesis  9:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  perceived  usefulness  of  C2IPS  and  trainee  performance  on  C2IPS. 

Hypothesis  12:  There  will  be  a  significant  positive  relationship  between  a 
trainee ’s  computer  attitude  and  trainee  performance  on  C2IPS. 

A  battery  of  stepwise  linear  regressions  was  performed  with  computer-self 
efficacy,  perceived  usefulness  of  C2IPS,  perceived  ease  of  use  of  C2IPS,  computer 
attitude,  and  self-reported  usage  of  C2IPS  using  self-rated  performance  as  the  dependent 
variable.  Regressions  used  the  combined  results  of  all  C2IPS  courses  (N=203)  based  on 
the  argument  developed  for  the  paired  t-test  analyses  above  (e.g.  each  training  course 
produced  similar  training  results).  Also,  self-rated  performance  was  used  as  the 
dependent  variables.  It  has  been  demonstrated  by  the  independent  t-test  between  self- 
rated  and  supervisor-rated  measures  of  performance  that  self-rated  performance  is  a 


64 


Table  16.  Linear  Regression  Results:  AMWC  C2IPS  Academic  Knowledge 
Test  (Learning)  as  Dependent  Variable 


Learning,  Time  1 

R2 

F 

Model  Statistics 

.267 

1.185 

Variable 

£ 

t 

Computer  Self-Efficacy,  Time  1 

.278 

1.088 

Perceived  Usefulness,  Time  1 

-.290 

-.861 

Perceived  Ease  of  Use,  Time  1 

.248 

.914 

Computer  Attitude,  Time  1 

.345 

1.020 

Learning,  Time  2, 

Model  1 


R2 

F 

Model  Statistics 

.191 

1.828 

Variable 

fi 

t 

Computer  Self-Efficacy,  Time  1 

.063 

.7*15 

Perceived  Usefulness,  Time  1 

.485 

.107 

Perceived  Ease  of  Use,  Time  1 

-.595 

.021 

Computer  Attitude,  Time  1 

-.117 

.680 

Learning,  Time  2, 

Model  2 

R2 

F 

Model  Statistics 

.033 

.2*56 

Variable 

fi 

t 

Computer  Self-Efficacy,  Time  2 

.096 

.604 

Perceived  Usefulness,  Time  2 

.056 

.830 

Perceived  Ease  of  Use,  Time  2 

-.132 

.601 

Computer  Attitude,  Time  2 

-.073 

.786 

reasonable  indicator  of  C2IPS  performance.  In  addition,  the  fact  that  the  quantity  of  self¬ 


rated  performance  measures  (N=207)  yields  more  degrees  of  freedom  and  statistical 
power  than  supervisor-report  performance  measures  (N=26)  supports  the  use  of  self-rated 
performance  measures.  Correlations  between  regression  variables  were  not  used  for  this 
analysis  but  nonetheless  are  contained  in  Appendix  C. 

As  shown  in  Table  17,  the  first  regression  used  beginning  of  course  (time  1)  self¬ 


rated  performance  as  the  dependent  variable  and  yielded  a  model  with  an  F-value  of 
32.61  (p  <  .001)  and  an  R-squared  value  of  .34.  Three  significant  predictors  were 


produced:  computer  self-efficacy,  perceived  ease  of  use,  and  self-reported  usage. 
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Perceived  usefulness  and  computer  attitude  were  not  significant.  Of  the  3  significant 
predictors,  computer  self-efficacy  had  the  highest  Beta- weight  at  .37  followed  by 
perceived  ease  of  use  (.24)  and  self-reported  usage  (.15).  All  predictors  were  significant 
at  the  p  <  .001  level. 

Next,  two  regression  models  were  produced  using  end  of  course  self-rated 
performance  (time  2)  as  the  dependent  variable.  The  first  model  used  time  1  independent 
variables  to  predict  performance  at  time  2.  Table  18  shows  that  the  model  was  significant 
(F  =  30.15,  p  <  .001).  The  R-squared  value  dropped  to  .23  as  compared  to  .34  at  time  1. 

Interestingly,  computer  self-efficacy  and  perceived  ease  of  use  still  showed  a 
capacity  to  predict  end-of  course  self-rated  performance.  The  next  regression  model 
utilized  time  2  independent  variables  to  predict  end  of  course  performance.  The  results 

Table  17.  Linear  Regression  Results  at  Time  1:  Beginning  of  Course  Self- 
Rated  Performance  as  Dependent  Variable 


AM  Courses  Combined  Using  Time  1  Independent  Variables 


Variable(s)  Included 
Computer  Self-Efficacy,  Time  1 
Perceived  Ease  of  Use,  Time  1 
Self-reported  usage 

Variable(s)  Excluded 
Perceived  Usefulness 
Computer  Attitude 


e! 

AR2 

F 

AF 

£ 

0.27 

0.27 

70.93*** 

70.93 

0.37 

0.31 

0.05 

44.37*** 

13.33 

0.24 

0.34 

0.02 

32.61*** 

6.54 

0.15 

_ 

_ 

_ 

.069 

- 

- 

- 

- 

.102 

t 

5.54* 

3.58* 

2.56* 


.995 

1.430 


*  Significant  at  p  <  .001 


were  almost  identical  to  the  beginning  of  course  regression  model  described  above.  As 


Table  18  shows,  the  model  was  significant  with  an  F-value  of  32.96  (p  <  .001).  The  R- 


squared  value  of  .34  was  identical  to  the  beginning  of  course  model.  In  addition, 


predictor  variables  were  identical:  perceived  ease  of  use  ((3  =  .37,  p  <  .001),  computer 
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self-efficacy  (P  =  .29,  p  <  .001),  and  self-reported  usage  (p  =  .13,  p  <  .001).  Similarly, 
computer  attitude  and  perceived  usefulness  failed  to  provide  significant  information. 

Table  18.  Linear  Regression  Results  at  Time  2:  End  of  Course  Self-Rated 
Performance  as  Dependent  Variable 


All  Courses  Combined  Using  Time  1  Independent  Variables 


Variable(s)  Included 

R2 

AR2 

F 

AF 

£ 

t 

Computer  Self-Efficacy,  Time  1 

0.18 

0.18 

42.41** 

42.41 

0.29 

4.12** 

Perceived  Ease  of  Use,  Time  1 

0.23 

0.06 

30.15** 

14.92 

0.27 

3.86** 

Variable(s)  Excluded 

Perceived  Usefulness,  Time  1 

- 

- 

- 

- 

-.029 

-.399 

Computer  Attitude 

- 

- 

- 

- 

.054 

.704 

Self-reported  usage 

- 

- 

- 

- 

.091 

1.413 

All  Courses  Combined  Using  Time  2  Independent  Variables 


Variablefs)  Included 

E! 

AR2 

F 

AF 

& 

t 

Perceived  Ease  of  Use,  Time  2 

0.25 

0.25 

63.81** 

63.81 

0.37 

5.86** 

Computer  Self-Efficacy,  Time  2 

0.32 

0.08 

46.32** 

21.94 

0.29 

4.59** 

Self-reported  usage 

0.34 

0.02 

32.96** 

4.54 

0.13 

2.13* 

Variablefs)  Excluded 

Perceived  Usefulness,  Time  2 

- 

- 

- 

- 

.078 

1.116 

Computer  Attitude 

- 

- 

- 

- 

.117 

1.440 

Significant  at  p  <  .05 
Significant  at  p  <  .001 


A  final  set  of  three  regressions  were  performed  using  on-the-job  self-rated 
performance  (time  3)  as  the  dependent  variable  (Table  19).  The  first  regression  used  time 
1  variables  to  predict  performance  and  was  significant  (F  =  8.44,  p  <  .001)  with  an 
overall  R2  of  .38.  In  this  model  computer  self-efficacy  (P  =  .36,  p  <  .1),  perceived 
usefulness  (P  =  .22,  p  <  .05),  and  self-reported  usage  (P  =  .23,  p  <  .05)  served  as 
significant  predictors  of  performance.  The  second  model  used  time  2  variables  as 

2 

predictors.  This  model  was  also  significant  with  an  F-value  of  10.3 1  (p  <  .001)  and  an  R 
=.33.  In  this  model,  end-of-course  self-rated  performance  (P  =  .31,  p  <  .1),  self-reported 
usage  (P  =  .32,  p  <  .05)  and  perceived  usefulness  (P  =  .25,  p  <  .05)  served  as  predictors. 
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Table  19.  Linear  Regression  Results  at  Time  3:  On-the-Job  Self-Rated 
Performance  as  Dependent  Variable 


AH  Courses  Combined  Using  Time  1  Independent  Variables 


Variablefs)  Included 

£ 

AR2 

F 

AF 

fi 

t 

Computer  Self-Efficacy,  Time  1 

.28 

.28 

17.12**** 

17.12 

.36 

2.51* 

Perceived  Usefulness,  Time  1 

.34 

.06 

10.83**** 

3.54 

.22 

1.69** 

Self-reported  usage 

.38 

.04 

8.44**** 

2.78 

.23 

1.67** 

Variablefs)  Excluded 

Perceived  Ease  of  Use,  Time  1 

- 

- 

_ 

_ 

-.05 

-.27 

Computer  Attitude 

- 

- 

- 

- 

.086 

.47 

All  Courses  Combined  Using  Time  2  Independent  Variables 


Variablefs)  Included 

E! 

AR2 

F 

AF 

£ 

t 

Self-Rated  Performance,  Time  2 

.26 

.26 

15.00**** 

15.00 

.31 

2.29* 

Self-reported  usage 

.35 

.21 

11.30*** 

5.90 

.32 

2.48** 

Perceived  Usefulness,  Time  2 

.41 

.12 

9.32**** 

3.90 

.25 

1.97* 

Variablefs)  Excluded 

Computer  Self-Efficacy,  Time  2 

- 

- 

- 

- 

-.026 

-.19 

Perceived  Ease  of  Use,  Time  2 

- 

- 

_ 

_ 

-.16 

.094 

Computer  Attitude 

- 

- 

- 

- 

-.20 

-1.25 

All  Courses  Combined  Using  Time  3  Independent  Variables 


Variablefs)  Included 

E! 

AR2 

F 

AF 

fi 

I 

Perceived  Usefulness,  Time  3 

.25 

.25 

14.27**** 

14.27 

.45 

3.82**** 

Self-reported  usage 

.41 

.16 

14.81**** 

11.84 

.41 

3.44**** 

Variablefs)  Excluded 

Perceived  Ease  of  Use,  Time  3 

- 

- 

- 

. 

.108 

.702 

Computer  Attitude,  Time  3 

- 

- 

- 

- 

-.082 

-.45 

Computer  Self-Efficacy,  Time  3 

- 

- 

- 

- 

.18 

1.51 

Significant  at  p  <  .1 
Significant  at  p  <  .05 
Significant  at  p  <  .01 
Significant  at  p  <  .001 


The  final  model  used  time  3  variables  as  predictors  of  performance  and  was  the 
most  statistically  significant  model  with  an  F-value  of  14.81  (p  <  .001)  and  R2  of  .41. 
Like  the  first  two  models,  perceived  usefulness  (P  =  .45,  p  <  .001)  and  self-reported 
usage  (p  =  .41,  p  <  .001)  proved  to  be  the  most  significant  indicators  of  performance. 

Of  note  is  the  fact  that  the  time  3  regressions  using  on-the-job  performance 
consistently  produced  perceived  usefulness  as  a  significant  variable  in  predicting 
performance.  On  the  other  hand,  time  1  and  time  2  regressions  using  beginning  and  end 
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of  course  performance  resulted  in  computer  self-efficacy  and  perceived  ease  of  use  as 
indicators  of  performance. 

Also  of  note  is  that  self-reported  usage  is  the  only  variable  that  significantly 
predicted  performance  during  all  times  of  the  study.  Given  the  guidelines  established  in 
Chapter  III,  this  consistency  in  the  ability  of  the  usage  variable  to  predict  performance 
better  at  different  times  enhances  its  reliability  as  a  predictor  of  performance. 

Table  20  shows  a  summary  of  the  linear  regression  models  and  significant 
predictor  variables  at  time  1,  time  2,  and  time  3.  Based  on  these  results,  hypotheses  3  and 
6  were  partially  supported  because  computer  self-efficacy  and  perceived  ease  of  use  were 
positively  related  to  performance  at  time  1  and  time  2  but  not  at  time  3.  Similarly, 
hypothesis  9  was  partially  supported  because  of  the  ability  of  perceived  usefulness  to 
predict  performance  at  time  3  but  not  at  time  1  and  time  2.  Finally,  hypothesis  12  was 
not  supported  due  to  the  inability  of  computer  attitude  to  predict  performance  at  any  time. 
These  findings  will  be  further  examined  in  the  following  chapter. 


Table  20.  Summary  of  Linear  Regression  Models  at  Times 

Time  1 

Time  2 

Time  3 

Performance 

Performance 

Performance 

Model  R2,  Time  1  Variables 

.34 

.23 

.38 

Model  R2,  Time  2  Variables 

.34 

.41 

Model  R2,  Time  3  Variables 

.41 

Variables 

fi 

r 

Computer  Self-Efficacy,  Time  1 

.37 

.29 

.36 

Computer  Self-Efficacy,  Time  2 

.29 

- 

Computer  Self-Efficacy,  Time  3 

- 

Perceived  Ease  of  Use,  Time  1 

.24 

.27 

- 

Perceived  Ease  of  Use,  Time  2 

.37 

- 

Perceived  Ease  of  Use,  Time  3 

- 

Perceived  Usefulness,  Time  1 

- 

- 

.23 

Perceived  Usefulness,  Time  2 

- 

.25 

Perceived  Usefulness,  Time  3 

.45 

Computer  Attitude,  Time  1 

- 

- 

- 

Computer  Attitude,  Time  2 

- 

- 

Computer  Attitude,  Time  3 

- 

Self-Reported  Usage,  Time  1 

.15 

- 

.23 

Self-Reported  Usage,  Time  2 

.13 

.32 

Self-Reported  Usage,  Time  3 

.41 
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Summary 


Chapter  IV  presented  the  results  and  analysis  of  the  data  compiled  for  this  study. 
Proposition  1  and  its  related  hypotheses  partially  supported  a  significant  increase  in 
trainee  reactions  and  performance  as  a  result  of  information  technology  training. 
Proposition  2  was  not  supported  meaning  that  a  significant  relationship  between  trainee 
reactions  and  learning  was  not  found.  Finally,  proposition  3  was  partially  supported. 
From  the  beginning  to  the  end  of  training,  computer  self-efficacy,  perceived  ease  of  use 
of  C2IPS,  and  self-reported  usage  of  C2IPS  were  significantly  related  to  C2IPS 
performance  increases.  For  on-the-job  performance,  perceived  usefulness  of  C2IPS,  and 
self-reported  usage  of  C2IPS  consistently  predicted  C2IPS  performance.  In  Chapter  V, 
the  impact  and  significance  of  these  findings  will  be  discussed  along  with  the  limitations 
of  the  study,  conclusions,  recommendations  based  on  the  findings,  and  implications  for 
future  practice  and  research. 
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V.  Discussion 


Chapter  Overview 

In  Chapter  IV,  results  and  analysis  of  this  study’s  data  was  presented.  Chapter  V 
will  now  discuss  the  significance  of  the  findings,  identify  limitations  of  the  research,  and 
offer  conclusions  and  recommendations.  In  addition,  implications  for  future  research  are 
outlined. 

Significance  of  Findings 

Proposition  1 .  There  will  be  a  significant  positive  increase  in  trainee  reactions 
following  C2IPS  training. 

The  key  insight  regarding  this  proposition  is  that  C2IPS  training  does  appear  to 
increase  trainee  reactions,  learning,  and  performance  during  training.  Further,  most  of 
these  increases  are  transferred  to  the  job.  Results  of  this  study  show  a  30%  increase  in 
computer  self-efficacy,  a  1 6%  increase  in  perceived  ease  of  use,  a  1 65%  increase  in 
learning,  and  a  47%  increase  in  performance  from  the  beginning  of  training  and  when  the 
trainee  resumes  his/her  normal  duties.  These  results  indicate  that  C2IPS  training  is 
successful  in  producing  more  effective  and  productive  C2IPS  operators  and  system 
administrators. 

Key  stakeholders  in  the  training  process  should  view  these  results  positively. 
Trainees  receive  crucial  skills  needed  to  perform  in  the  workplace,  and  supervisors  in  Air 
Mobility  Command  field  organizations  receive  more  productive  personnel.  Supervisors 
can  use  these  variables  to  assess  whether  or  not  an  individual  should  be  sent  to  training 
and  as  a  measure  of  performance  improvement  after  training.  Air  Education  and  Training 
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Command  training  instructors  can  be  confident  that  their  training  curriculum  and 
techniques  are  effective  in  producing  skilled  C2IPS  personnel.  Increases  and  decreases 
in  variables  could  provide  an  indication  of  trainee  progress  and  notify  decision-makers  of 
the  impact  of  C2IPS  training  on  performance.  Air  Mobility  Command  training  resource 
providers  can  use  these  impacts  to  assess  whether  funding,  staff,  and  facilities 
investments  are  yielding  returns  and  to  establish  investment  criteria.  This  investment 
could  include  the  development  of  new  information  technology  training  methods  by  the 
Air  Force  Research  Laboratory.  By  using  the  surveys  developed  for  this  study,  the  Air 
Force  research  laboratory  could  assess  whether  techniques  for  increasing  training 
performance  using  a  new  system  actually  work  before  recommending  the  system  future 
acquisition.  For  the  Air  Force,  then,  the  cost  of  C2IPS  training  is  worth  the  benefit 
received  in  the  form  of  support  to  "Global  Reach"  and  "Information  Dominance". 

Although  some  variables  can  be  used  to  great  effect  by  supervisors,  training 
instructors,  and  resource  providers,  use  of  perceived  usefulness  and  computer  attitude 
should  be  viewed  with  caution.  These  variables  returned  to  their  original  beginning  of 
course  levels  despite  the  increase  attributed  to  C2IPS  training.  From  an  academic 
standpoint,  the  resiliency  of  trainee  usefulness  perceptions  and  computer  attitudes  may 
indicate  that  these  constructs  are  more  closely  related  to  value  judgements  that  have  been 
established  over  a  long  period  of  exposure  to  computer  systems  like  C2IPS.  According 
to  the  Theory  of  Reasoned  Action  (Fishbein  and  Aijzen,  1974),  value  judgments  are  less 
likely  to  be  influenced  by  temporary  interventions  such  as  training.  In  addition, 
perceived  usefulness  and  computer  attitude  may  encompass  a  broader  view  of  the  work 
context.  Since  C2IPS  training  is  focused  on  a  narrow  aspect  of  the  trainee’s  work 
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domain,  the  training  may  “tap  into”  only  a  part  of  the  entire  usefulness  and  attitude 
constructs  as  viewed  by  the  trainee.  In  addition,  the  job  environment  may,  in  fact,  be  the 
place  where  the  trainee  derives  the  strongest  support  and  cues  for  his/her  usefulness 
perceptions  and  computer  attitudes.  These  may,  in  turn,  “override”  the  training  increases 
and  cause  the  trainee  to  return  to  his/her  original  levels. 

Proposition  2.  The  better  a  trainee’s  reactions  to  IT  training,  the  higher  the 
trainee’s  learning. 

None  of  the  reaction  variables  supported  a  relationship  to  trainee  learning.  A 
couple  of  explanations  may  clarify  why  this  result  occurred.  First,  the  sample  size  may 
have  been  too  low  to  detect  any  effects  due  to  only  one  training  course  utilizing  C2IPS 
academic  knowledge  testing.  Second,  there  may  have  been  no  relation  between  the 
reaction  variables  and  the  material  tested.  At  a  minimum,  computer  self-efficacy  should 
have  predicted  trainee  learning  due  to  the  strong  literature  supporting  the  self-efficacy-to- 
leaming  link.  Perhaps,  repeating  this  part  of  the  study  with  a  larger  sample  size  and  a 
more  detailed  review  of  test  materials  may  yield  significant  results.  Even  though  this 
relationship  was  not  supported  for  this  study,  it  should  not  be  dismissed  as  unimportant. 
At  the  least,  training  instructors  should  request  more  thorough  investigation  into  the 
learning-performance  relationship. 

Proposition  3.  The  better  a  trainee’s  reactions  to  IT  training,  the  better  the 
trainee’s  on-the-job  performance. 

As  observed  in  Chapter  IV,  some  variables  predict  performance  better  at  different 
times.  Performance  associated  with  training  (beginning  and  end  of  course  self-rated 
performance  in  this  case)  are  predicted  better  by  computer  self-efficacy  and  perceived 
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ease  of  use.  Conversely,  on-the-job  performance  is  predicted  better  by  perceived 
usefulness.  Computer  attitude  was  not  effective  in  predicting  performance  at  all,  but  self- 
reported  usage  was  the  only  variable  effective  in  predicting  both  training  and  on-the-job 
performance.  Before  discussing  the  practical  importance  of  these  results,  the  disparity  in 
the  ability  of  variable  to  predict  performance  at  different  times  deserves  discussion. 

As  Table  19  showed,  computer  self-efficacy,  perceived  ease  of  use,  and  self- 
reported  usage  were  relatively  stable  in  explaining  the  variance  associated  with 
performance  throughout  C2IPS  training.  The  fact  that  both  the  time  1  and  time  2 
regression  models  had  R2  values  of  .34  also  contributes  to  the  argument  of  stability. 
Clearly  computer  self-efficacy  and  perceived  ease  of  use  are  related  to  performance  using 
C2IPS  during  training.  For  computer  self-efficacy,  previous  results  from  the  self-efficacy 
literature  seem  to  have  been  borne  out  by  this  study,  especially  with  regard  to  increasing 
a  person’s  confidence  and  mobilizing  behavior  to  attain  performance  (Locke,  et  al.,  1984; 
Wood  and  Bandura,  1 989).  Similarly,  the  hypothesized  but  never  demonstrated 
perceived  ease  of  use  to  performance  link  has  been  supported  (Davis,  1989).  In  fact,  the 
ability  of  perceived  ease  of  use  to  predict  performance  from  the  beginning  of  training  to 
the  end  of  training  actually  increased.  This  increase  could  be  explained  through  the 
increased  effectiveness  of  thought  processes,  manipulations,  and  procedures  required 
effectively  use  the  information  system  as  a  result  of  simply  progressing  through  the 
training  course.  For  example,  a  procedure  to  enter  information  into  a  database  that  once 
may  have  seemed  daunting  or  impossible  before  training  may  seem  easy  once  taught  to 
and  practiced  by  the  trainee. 
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When  viewed  from  a  post-training,  on-the-job  standpoint,  the  performance  picture 
changes.  As  shown  in  Table  19,  perceived  usefulness  and  self-reported  usage  become 
significant  predictors  of  on-the-job  performance  with  computer  self-efficacy  significant 
only  at  time  1 .  In  fact,  the  ability  of  perceived  usefulness  to  predict  performance 
improves  over  time.  When  connected  with  the  definition  of  perceived  usefulness,  the 
behavior  of  perceived  usefulness  is  more  easily  explained.  According  to  Davis  (1989), 
perceived  usefulness  is  defined  as  the  prospective  user’s  subjective  probability  that  using 
an  information  system  will  increase  his  job  performance  within  an  organizational  context. 
Since  the  training  for  this  study  was  conducted  outside  the  trainee’s  normal  work 
environment,  it  seems  plausible  that  the  “organizational  context”  component  of  perceived 
usefulness  was  rendered  irrelevant.  In  addition,  the  focus  of  C2IPS  training  was  to  learn 
to  use  the  system  rather  than  to  figure  out  how  the  system  would  contribute  to  job 
performance.  Finally,  in  a  typical  work  environment,  job  performance  is  the  primary 
focus  of  daily  activity.  In  this  environment,  the  relevance  of  C2IPS  in  attaining  the 
required  performance  is  likely  to  be  a  much  more  germane  issue  for  most  Air  Force 
members.  In  the  job  environment,  perceived  usefulness  likely  has  more  relevance  as  a 
measure  of  on-the-job  performance.  Using  these  explanations,  it  seems  reasonable  to 
assume  that  the  link  between  perceived  usefulness  and  on-the-job  performance  would  be 
weak  at  the  beginning  of  training  when  the  trainee  has  just  arrived  in  the  training 
environment.  At  the  end  of  training,  perceived  usefulness  would  be  higher  as  the  trainee 
begins  to  make  the  connection  between  what  he/she  has  learned  and  how  it  may 
contribute  to  the  job  context.  Finally,  perceived  usefulness  would  be  highest  on-the-job 
where  the  use  of  the  system  and  job  performance  are  more  easily  linked. 
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Self-reported  usage  was  the  “surprise”  variable  of  this  study.  Hypotheses 
regarding  self-reported  usage  were  not  made  for  this  study,  but  its  efficacy  in  predicting 
performance  across  all  stages  of  the  training  continuum  deserves  some  discussion. 
Intuitively,  it  seems  that  the  more  a  person  uses  an  information  system,  the  better  he  or 
she  should  perform.  Therefore,  a  relationship  between  usage  and  performance  is  entirely 
feasible.  Indeed,  the  results  presented  in  Chapter  IV  bear  this  out.  Use  of  a  system  may 
be  described  as  a  necessary  but  not  sufficient  component  of  performance  along  with  other 
components  such  as  level  and  duration.  Despite  the  promise  of  this  explanation,  the 
relationship  is  anything  but  conclusive  due  to  the  design  of  this  study.  For  instance,  self- 
reported  usage  was  measured  only  once  at  the  beginning  of  the  training  courses  using  one 
item.  The  item  asked  how  much  time  the  trainee  spent  on  C2IPS  in  a  given  day. 

Repeated  measures  and  more  items  that  capture  self-reported  usage  may  increase  the 
validity  of  the  measure  somewhat.  Better  yet,  actual  measures  of  usage  would  provide 
even  more  credibility.  Nonetheless,  if  one  conceives  of  self-reported  usage  as  an 
“exploratory”  variable,  then  the  consistent  results  it  produced  certainly  merits  more  in- 
depth  investigation  as  to  the  strength  and  nature  of  the  usage  to  performance  relationship. 

Computer  attitude  was  not  significant  in  predicting  performance  at  any  time.  The 
most  likely  explanation  is  that  computer  attitude  as  measured  is  “broad  band”  and 
perhaps  captured  attitudes  about  computers  in  general  rather  than  those  associated  strictly 
with  C2IPS.  In  this  sense,  the  computer  attitude  measure  may  not  have  been  focused 
enough  to  capture  a  relationship  with  performance.  In  addition,  in  several  cases 
computer  attitude  was  nearly  significant  suggesting  that  larger  numbers  of  participants 
may  have  yielded  a  significant  result.  Whether  a  larger  sample  size  correlates  to 
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measuring  a  “broad  band”  relationship  of  computer  attitude  to  performance  is  debatable. 
At  best,  supervisors  and  training  instructors  should  be  cautioned  in  their  use  of  computer 
attitude  as  a  predictor  of  performance  and  utility  of  C2IPS. 

Based  on  the  results  and  observations  presented  above,  a  revised  information 
technology  training  model  can  be  formulated.  This  model  is  shown  in  Figure  8. 


REVISED  INFORMATION  TECHNOLOGY 
TRAINING  EVALUATION  MODEL 


Figure  8.  Revised  Information  Technology  Training  Model 

The  fact  that  some  variables  influence  performance  can  be  of  practical  use  to 
training  instructors.  Knowing  which  variables  predict  performance  enables  instructors  to 
develop  and  improve  methods  to  increase  performance  that  target  the  desired  variable.  In 
this  way,  the  training  course  can  be  focused  to  on  the  variables  that  will  maximize 
training  performance.  In  addition,  instructors,  supervisors,  and  resource  providers  can 
use  these  variables  as  surrogates  of  performance  when  measurement  of  performance  is 
impractical  or  difficult. 
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Perhaps  the  most  significant  use  of  predictor  variables  is  in  the  area  of 
information  technology  training  development.  The  development  of  a  new  information 
system  entails  many  uncertainties  including  the  ability  of  a  user  to  perform  job  task  using 
the  system.  In  addition,  the  ability  of  the  system  itself  to  facilitate  performance  of 
specified  tasks  may  be  uncertain.  By  measuring  a  user's  computer  self-efficacy, 
perceived  usefulness,  and  perceived  ease  of  use,  a  reasonable  indication  of  the  user's 
ability  to  perform  may  be  produced.  Low  measures  may  indicate  deficiencies  in  training 
methods  or  performance  criteria.  These  measures  in  turn  can  provide  feedback  to 
developers  for  improving  both  the  information  system  and  the  training  on  the  system 
before  investing  capital  toward  future  development. 

Additional  Observations 

The  fact  that  the  means  for  on-the-job  self-rated  performance  and  supervisor-rated 
performance  were  exactly  the  same  was  unexpected.  Given  the  literature  on  the  subject, 
self-rated  performance  was  expected  to  be  significantly  higher  than  supervisor-rated 
performance.  One  possible  explanation  for  this  is  that  the  training  may  have  taken  place 
recently  enough  for  both  supervisor  and  subordinate  to  make  sufficiently  unbiased 
estimates  of  C2IPS  performance.  Additionally,  the  performance  instrument  itself  may 
have  been  narrow  enough  to  eliminate  any  inflationary  bias.  Despite  the  positive  impact 
of  this  observation  on  this  study,  however,  the  small  number  of  cases  analyzed  (N=22) 
makes  it  unclear  whether  the  results  can  be  generalized  to  a  larger  population. 

From  a  practical  standpoint,  however,  supervisors,  instructors,  and  training 
resource  providers  may  view  self-rated  performance  measures  as  "suitable"  substitutes 
for  more  "objective"  supervisor  ratings. 
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Limitations  of  Study 


Several  factors  limit  the  generalizability  of  this  study  including  the  research 
design,  sample  population,  sample  size,  performance  measure  used,  and  the  limited  focus 
of  this  study  centering  on  learning  and  performance.  No  study  can  be  perfect,  and 
improvements  or  changes  in  these  areas  may  lead  to  different  results  and  conclusions. 

The  research  design  for  this  study  used  a  variation  of  the  quasi-experimental  time 
series  design.  In  a  time  series  design,  several  measures  are  taken  before  and  after  an 
event  and  allows  subjects  to  be  their  own  controls  (Cooper  and  Emery,  1995).  However, 
the  main  internal  validity  problem  for  this  design  is  history.  History,  in  the  case  of  this 
study,  is  the  chance  that  events  could  have  occurred  that  confused  the  relationship  of 
performance  and  learning  to  the  reaction  variables.  Since  each  training  class  was 
conducted  at  different  times  and  places  without  the  presence  of  trained  researchers,  such 
events  could  likely  have  occurred.  In  addition,  maturation  and  testing  effects  could  have 
had  an  influence  on  the  participants.  Testing  effects  are  a  particular  concern  as  all 
variables  were  measured  a  total  of  3  different  times  using  exactly  the  same  items  and 
scales.  An  alternative  approach  may  be  to  develop  several  items  for  each  measure  and 
construct  multiple  surveys  and  randomize  items  to  reduce  such  effects.  In  addition,  self- 
reported  usage  was  measured  only  once  at  the  beginning  of  training  and  supervisor-rated 
performance  was  measured  only  once  on-the-job.  Additional  measures  of  these  variables 
would  provide  a  basis  of  comparison  and  eliminate  some  internal  validity  concerns.  It  is 
hoped  that  the  measurement  of  multiple  training  classes  may  have  diluted  these  effects 
somewhat,  but  the  reality  is  that  the  data  collected  for  this  study  may  have  been  adversely 
impacted  in  unpredictable  ways. 
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External  validity  is  also  limited  in  this  study  by  virtue  of  the  fact  that  a 
convenience  sample  representing  a  narrow  part  of  the  Air  Force  population  was  used. 

The  results  may  be  extremely  valid  for  Air  Force  C2IPS  personnel,  but  may  not  be 
applicable  for  other  groups  who  undergo  information  technology  training,  for  example,  in 
a  commercial  software  firm.  Repeated  application  of  the  hypothesis  procedures  used  in 
this  study  on  other  sample  populations  will  enhance  the  general  validity  of  this  study’s 
results. 

A  larger  sample  size  for  each  training  course  would  not  only  increase  the  overall 
statistical  power  of  the  study,  but  also  would  allow  comparisons  between  training  courses 
and  better  detection  of  the  similarities  and  differences  between  them.  For  example, 
potential  analysis  of  C2IPS  demographic  effects  was  largely  precluded  due  to  the 
relatively  small  sample  sizes  associated  with  each  training  course. 

The  performance  measure  used  for  analyzing  data  was  subjective.  Even  though  a 
case  was  made  for  the  use  of  self-reports  in  Chapter  II,  an  objective  performance  measure 
would  likely  be  more  credible  for  both  the  researcher  and  the  practitioner.  Even  so,  the 
scope,  duration,  and  window  of  opportunity  of  this  study  precluded  the  collection  of  such 
a  measure. 

The  last  major  limitation  concerns  the  variance  explained  in  the  relationship 
between  reactions  and  leaming/performance.  Clearly,  the  models  do  not  explain  all  that 
accounts  for  increases  in  performance  and  learning.  Even  more  clearly,  many  more 
aspects  of  training  can  influence  learning  and  performance  such  as  the  training  instructor, 
training  environment,  training  method,  training  quality,  personal  factors,  supervisor 
support,  and  work  environment.  Some  or  all  of  these  factors  may  contribute  significantly 
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to  increasing  learning  and  performance  as  the  result  of  information  technology  training. 
Thus,  additional  research  could  focus  on  extending  this  study  to  include  other  variables. 

Conclusions  and  Recommendations 

Despite  limitations  of  this  study,  several  reasonable  conclusions  can  be  offered 
with  regard  to  C2IPS  training. 

-  Information  technology  training  increases  computer  self-efficacy,  perceived 
ease  of  use,  learning,  and  performance. 

-  These  increases  are  transferred  to  the  work  environment  resulting  in  a  more 
confident  and  proficient  user  of  C2IPS. 

Computer  self-efficacy,  perceived  ease  of  use,  perceived  usefulness,  and  self- 
reported  usage  are  significant  predictors  of  training  performance  on  C2IPS. 

-  Computer  attitude  does  not  predict  performance  at  any  time. 

From  a  practical  standpoint,  supervisors  can  use  computer  self-efficacy  and 
perceived  ease  of  use  as  quick  indicators  of  performance  in  lieu  of  objective  measures  of 
performance  and  to  evaluate  a  trainee’s  readiness  or  need  to  attend  training.  If  a  person’s 
computer  self-efficacy  and  perceived  ease  of  use  is  low,  then  performance  may  be  low, 
indicating  a  need  for  training.  Training  instructors  can  use  the  measures  to  evaluate  a 
trainee’s  disposition  toward  performance  before  starting  training  and  their  performance 
progress  during  training.  Similarly,  perceived  usefulness  can  be  used  as  an  indicator  of 
future  performance  after  training  or  current  performance  on  a  particular  information 
system  and  be  used  to  explain  how  the  information  system  actually  contributes  to  job 
performance.  Self-reported  usage  may  be  the  “ace  in  the  hole”  for  both  trainer  and 
supervisor.  Because  usage  seems  to  impact  performance  both  in  training  and  on-the-job, 
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getting  an  individual  working  on  a  system  is  a  definite  way  to  begin  increasing 
performance.  Finally,  information  technology  developers  and  training  developers  can  use 
the  predictor  variables  decribed  in  this  study  as  indicators  of  whether  or  not  a  particular 
system  or  training  program  contributes  to  performance.  Judicious  use  of  these  variable 
may  reduce  uncertainty  in  and  improve  future  system  and  training  development. 

Overall,  the  C2IPS  training  program  seems  to  be  providing  highly  motivated, 
trained  users  who  increase  and  sustain  their  performance  on  C2IPS  upon  return  to  their 
jobs.  On-going  collection  of  data  similar  to  that  presented  in  this  study  can  ensure  this 
success  story  continues.  Because  of  the  stability  of  the  measures  over  time, 
administration  of  surveys  both  during  training  and  on-the-job  will  yield  valid  data  on 
computer  self-efficacy,  perceived  ease  of  use,  perceived  usefulness,  self-reported  usage, 
and  performance.  This  data  will  provide  assurance  to  C2IPS  instructors,  training 
managers,  and  field  supervisors  that  C2IPS  training  courses  are  meeting  their  goals.  In 
addition,  this  data  provides  valuable  information  that  developers  and  senior  decision¬ 
makers  need  to  continue  to  improve  and  fund  the  C2IPS  training  program. 

Implications  for  Future  Research 

Based  on  this  study,  several  interesting  avenues  for  future  research  exist. 
Generalizability  of  the  reaction  to  performance  relationships  described  above  would  be 
increased  if  other  groups  and  training  courses  were  measured.  The  study  of  the 
information  technology  usage  to  performance  relationship  using  either  subjective  or 
objective  measures  would  shed  light  on  the  exact  nature  of  the  influence  of  usage  on 
performance  and  what  other  factors  may  contribute  to  performance.  The  inelasticity  of 
perceived  usefulness  and  computer  attitude  to  training  interventions  could  help  explain 
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how  these  perceptions  are  formed  and  how  they  might  be  changed  in  a  more  lasting  way, 
especially  when  the  relationship  of  perceived  usefulness  to  performance  is  taken  into 
account.  Next,  more  study  is  warranted  regarding  the  influence  of  reactions  on  learning. 
The  literature  supports  this  link,  but  the  link  was  not  supported  by  this  study.  Also  useful 
would  be  an  investigation  as  to  the  influence  of  the  work  environment  on  performance 
using  information  technology.  Finally,  a  comparison  of  formal  information  technology 
training  like  that  used  for  this  study  against  other  training  methods  such  as  on-the-job 
training  could  help  show  if  one  method  is  better  than  the  other  using  differences  in 
variable  predictability  as  gauges. 

Summary 

Information  technology  training  is  crucial  for  providing  personnel  the  requisite 
skills  needed  to  perform  effectively  if  the  Air  Force  is  to  remain  competitive  in  today’s 
automated  warfighting  environment.  Information  technology  training  programs  that 
continuously  provide  these  skilled  personnel  will  be  the  ones  that  are  embraced, 
supported,  and  sustained  by  the  Air  Force  despite  dwindling  resources  to  allocate  toward 
these  efforts.  This  study  explored  one  such  training  program  in  an  attempt  to  capture  key 
elements  that  influence  information  technology  performance  both  during  training  and  on- 
the-job.  An  understanding  of  the  magnitude,  relationships,  and  nature  of  these  elements 
will  help  training  instructors  design  and  deliver  higher  quality  training,  enable 
supervisors  to  identify  people  for  training  and  gauge  their  performance  after  training,  and 
enable  decision  makers  to  allocate  scarce  resources  to  information  technology  training 
programs  that  produce  results.  For  the  Air  Force,  effective  information  technology 
training  programs  are  essential  achieving  “Information  Dominance”  and  “Global  Reach”. 
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Normal  Q-Q  Plot  of  On-the-Job  Computer  Self-Efficacy 


Perceived  Usefulness 


Beginning  of  Course 


Std.  Dev  =  5.84 
Mean  =  20.1 
N  =  207.00 
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Normal  Q-Q  Plot  of  Beginning  of  Course  Perceived  Usefulness 
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End  of  Course  Perceived  Usefulness 


Norma!  Q-Q  Plot  of  End  of  Course  Perceived  Usefulness 


Std.  Dev  =  5.71 
Mean  =  22.4 
N  =  204.00 
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Norma!  Q-Q  Plot  of  On-the-Job  Perceived  Usefulness 


Perceived  Ease  of  Use 


Normal  Q-Q  Plot  of  Beginning  of  Course  Perceived  Ease  of  Use 
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End  of  Course 


Std.  Dev  =  6.08 
Mean  =  20.1 
N  =  204.00 
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Normal  Q-Q  Plot  of  End  of  Course  Perceived  Ease  of  Use 
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Computer  Attitude 


Beginning  of  Course 


Std.  Dev  =  5.30 
Mean  =  19.9 
N  =  206.00 
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Normal  Q-Q  Plot  of  Beginning  of  Course  Computer  Attitude 
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Std.  Dev  =  4.95 
Mean  =  22.1 
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Normal  Q-Q  Plot  of  End  of  Course  Computer  Attitude 
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Std.  Dev  =  4.82 
Mean  =  21.0 
N  =  52.00 
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Normal  Q-Q  Plot  of  On-the-Job  Computer  Attitude 
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Self-Rated  Performance 
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Std.  Dev  =  1.17 
Mean  =  2.67 
N  =  203.00 
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Normal  Q-Q  Plot  of  Beginning  of  Course  Self-Rated  Performance 
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Appendix  B:  Overall  Factor  Analysis  for  Scale  Variables 


TIME  1  AND  TIME  2 

Communalities.  Time  2 

Computer  Self-Efficacy  Item  1  1  1 

Computer  Self-Efficacy  Item  2  1  1 

Computer  Self-Efficacy  Item  3  1  I 

Computer  Self-Efficacy  Item  4  1  1 

Computer  Self-Efficacy  Item  5  1  1 

Computer  Self-Efficacy  Item  6  I  1 

Computer  Self-Efficacy  Item  7  1  1 

Computer  Self-Efficacy  Item  8  1  1 

Computer  Self-Efficacy  Item  9  1  I 

Computer  Self-Efficacy  Item  10  1  1 

Perceived  Usefulness  Item  1  1  1 

Perceived  Usefulness  Item  2  I  1 

Perceived  Usefulness  Item  3  1  1 

Perceived  Usefulness  Item  4  1  \ 

Perceived  Ease  Of  Use  Item  1  1  1 

Perceived  Ease  Of  Use  Item  2  1  1 

Perceived  Ease  Of  Use  Item  3  I  1 

Perceived  Ease  Of  Use  Item  4  1  1 

Computer  Attitude  Item  1  1  1 

Computer  Attitude  Item  2  1  1 

Computer  Attitude  Item  3  1  ] 

Computer  Attitude  Item  4  1  1 

Self-Rated  Performance  Item  1  1  1 

Self-Rated  Performance  Item  2  1  1 

Self-Rated  Performance  Item  3  I  1 

Self-Rated  Performance  Item  4  1  1 

Self-Rated  Performance  Item  5  I  1 

Self-Rated  Performance  Item  6  1  1 

Self-Rated  Performance  Item  7  I  1 

Self-Rated  Performance  Item  8  1  1 

Self-Rated  Performance  Item  9  I  1 

Self-Rated  Performance  Item  10  1  1 

Self-Rated  Performance  Item  11  1  1 

Self-Rated  Performance  Item  12  1  1 

Self-Rated  Performance  Item  13  1  1 

Self-Rated  Performance  Item  14  1  1 

Self-Rated  Performance  Item  15  1  1 

Self-Rated  Performance  Item  16  1  1 

Self-Rated  Performance  Item  17  1  1 

Self-Rated  Performance  Item  18  1  1 

Self-Rated  Performance  Item  19  1  1 

Self-Rated  Performance  Item  20  1  1 

Self-Rated  Performance  Item  21  1  1 

Self-Rated  Performance  Item  22  1  1 

Self-Rated  Performance  Item  23  1  1 


COMMUNALITIES  USING  VARIMAX  ROTATION, 

Item  Communalities.  Time  1 
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OVERALL  FACTOR  ANALYSIS  ROTATED  COMPONENT  MATRIX,  TIME  1 


Component  1 

Component  2 

Component  3 

Component  4 

Component  5 

Component  6 

034 

0.49 

0.02 

0.12 

0.28 

0.53 

0.16 

0.65 

-0.04 

0.00 

0.20 

0.04 

0.28 

0.68 

0.08 

0.07 

0.20 

-0.04 

0.28 

0.80 

-0.06 

-0.02 

0.18 

-0.04 

0.26 

0.83 

0.04 

0.09 

0.05 

0.08 

0.26 

0.87 

0.05 

0.08 

0.03 

0.09 

0.18 

0.86 

0.08 

0.04 

0.03 

0.12 

0.29 

0.76 

0.14 

0.01 

0.13 

0.13 

0.21 

0.83 

0.12 

0.13 

-0.09 

0.05 

0.10 

0.75 

0.14 

0.12 

0.13 

0.06 

0.04 

0.12 

0.25 

0.82 

0.17 

0.10 

0.16 

0.10 

0.22 

0.88 

0.12 

-0.04 

0.17 

0.10 

0.36 

0.80 

0.20 

0.01 

0.21 

0.07 

0.47 

0.71 

0.17 

0.11 

0.23 

0.28 

0.05 

0.28 

0.50 

0.38 

0.18 

0.16 

0.11 

0.17 

0.76 

0.04 

0.25 

0.25 

0.27 

0.24 

0.76 

0.08 

0.19 

0.19 

0.41 

0.12 

0.72 

0.07 

0.15 

0.11 

0.84 

0.37 

0.09 

0.02 

0.16 

0.11 

0.86 

0.30 

0.09 

0.04 

0.12 

0.05 

0.83 

0.28 

0.23 

0.06 

0.10 

0.10 

0.65 

0.19 

0.49 

0.02 

0.70 

0.24 

0.09 

0.11 

0.19 

0.42 

0.78 

0.16 

0.17 

0.04 

0.12 

0.42 

0.70 

0.18 

0.20 

0.02 

0.23 

0.52 

0.73 

0.22 

0.20 

0.03 

0.19 

0.47 

0.77 

0.24 

0.10 

0.09 

0.10 

0.48 

0.81 

0.20 

-0.01 

0.11 

0.04 

0.31 

0.82 

0.26 

0.10 

0.08 

0.08 

0.29 

0.85 

0.14 

0.11 

0.06 

0.15 

-0.19 

0.90 

0.18 

0.11 

0.01 

0.10 

0.05 

0.91 

0.16 

0.08 

0.04 

0.13 

-0.02 

0.91 

0.17 

0.07 

0.08 

0.15 

-0.04 

0.91 

0.18 

0.09 

0.06 

0.14 

0.01 

0.88 

0.10 

0.13 

0.07 

0.15 

-0.18 

0.89 

0.19 

-0.01 

0.07 

0.04 

0.18 

0.88 

0.16 

-0.04 

0.12 

0.04 

0.08 

0.80 

0.23 

0.13 

0.07 

0.20 

-0.01 

0.87 

0.19 

0.11 

0.05 

0.08 

0.15 

0.82 

0.27 

0.12 

0.00 

0.14 

-0.05 

0.89 

0.19 

-0.03 

0.11 

0.07 

-0.08 

0.71 

0.24 

0.17 

0.09 

0.16 

0.11 

0.86 

0.22 

0.09 

0.12 

0.02 

0.08 

0.84 

0.22 

0.04 

0.16 

-0.02 

0.08 

0.80 

0.25 

0.07 

0.16 

-0.03 

0.28 
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OVERALL  FACTOR  ANALYSIS  ROTATED  COMPONENT  MATRIX,  TIME  2 

Component  I  Component  2  Component  3  Component  4  Component  5  Component  6  Component  7 


0.23 

0.53 

0.20 

0.26 

0.16 

0.06 

-0.03 

0.07 

0.69 

0.00 

0.10 

0.16 

0.05 

0.31 

0.07 

0.72 

0.01 

0.11 

0.13 

0.02 

0.29 

0.11 

0.77 

0.09 

0.09 

0.08 

-0.03 

0.13 

0.11 

0.87 

0.07 

0.08 

0.08 

0.03 

-0.01 

0.12 

0.89 

0.11 

0.06 

0.04 

0.03 

-0.07 

0.08 

0.86 

0.17 

0.04 

0.03 

-0.02 

-0.02 

0.19 

0.74 

0.17 

0.00 

0.12 

-0.02 

0.17 

0.11 

0.80 

0.09 

0.18 

0.04 

0.00 

-0.13 

0.17 

0.79 

0.20 

0.06 

0.04 

0.00 

-0.14 

0.09 

0.19 

0.27 

0.06 

0.86 

-0.04 

0.03 

0.16 

0.15 

0.27 

0.07 

0.84 

-0.06 

0.06 

0.07 

0.15 

0.26 

0.09 

0.87 

-0.04 

0.03 

0.15 

0.09 

0.30 

0.10 

0.85 

-0.05 

0.05 

0.21 

0.18 

0.53 

0.22 

0.45 

-0.08 

0.11 

0.12 

0.19 

0.47 

0.06 

0.07 

-0.18 

0.58 

0.21 

0.18 

0.78 

0.16 

0.06 

-0.04 

0.28 

0.26 

0.10 

0.73 

0.12 

0.22 

-0.02 

0.06 

0.06 

0.21 

0.71 

0.10 

0.37 

0.19 

-0.22 

0.08 

0.14 

0.77 

0.11 

0.37 

0.16 

-0.20 

0.06 

0.18 

0.75 

0.11 

0.28 

0.12 

-0.02 

0.17 

0.13 

0.71 

0.12 

0.15 

0.07 

0.19 

0.28 

0.12 

0.14 

0.73 

0.09 

0.10 

-0.02 

0.49 

0.16 

0.13 

0.74 

0.10 

0.05 

0.10 

0.43 

0.21 

0.16 

0.77 

0.07 

0.04 

0.07 

0.48 

0.22 

0.20 

0.73 

0.11 

0.07 

0.07 

0.54 

0.19 

0.19 

0.68 

0.09 

0.06 

0.11 

0.59 

0.09 

0.15 

0.65 

0.08 

0.03 

-0.07 

0.63 

0.11 

0.14 

0.50 

0.15 

0.10 

0.21 

0.80 

0.10 

0.05 

0.07 

0.09 

-0.09 

0.19 

0.87 

0.10 

0.17 

0.19 

0.00 

0.04 

-0.06 

0.89 

0.05 

0.17 

0.18 

-0.01 

-0.01 

-0.08 

0.91 

0.08 

0.16 

0.16 

0.04 

-0.05 

-0.08 

0.87 

0.13 

0.15 

0.19 

-0.01 

-0.06 

-0.06 

0.81 

0.16 

0.11 

-0.01 

0.18 

-0.04 

0.14 

0.70 

0.12 

0.19 

0.44 

0.02 

0.07 

-0.08 

0.74 

0.16 

0.10 

0.31 

0.08 

0.12 

-0.17 

0.61 

0.15 

0.21 

0.41 

0.03 

0.15 

0.32 

0.71 

0.08 

0.09 

0.31 

0.17 

0.19 

0.23 

0.67 

0.17 

0.04 

0.22 

0.13 

0.10 

0.37 

0.82 

0.18 

0.05 

0.18 

0.09 

0.12 

0.04 

0.47 

0.23 

0.06 

0.35 

0.15 

0.09 

0.44 

0.75 

0.14 

-0.04 

0.26 

0.14 

0.24 

0.10 

0.11 

-0.01 

0.08 

0.07 

-0.07 

0.88 

-0.05 

0.12 

0.04 

0.12 

0.14 

-0.09 

0.89 

0.01 
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OVERALL  FACTOR  ANALYSIS  TOTAL  VARIANCE  EXPLAINED,  TIME  1 

(CONVERGED  IN  7  ITERATIONS) 


Initial  Eigenvalues  Rotation  of  Sums  of  Squares  Loadings 


Component 

Total 

%  of  Variance 

Cumulative  % 

Total 

%  of  Variance 

Cumulative 

1.00 

22.17 

49.26 

49.26 

16.80 

37.33 

37.33 

2.00 

5.39 

11.98 

61.24 

7.02 

15.59 

52.92 

3.00 

4.12 

9.15 

70.39 

3.63 

8.08 

61.00 

4.00 

1.62 

3.61 

74.00 

3.34 

7.42 

68.41 

5.00 

1.32 

2.93 

76.93 

2.93 

6.51 

74.93 

6.00 

1.12 

2.50 

79.42 

2.02 

4.49 

79.42 

7.00 

0.85 

1.89 

81.31 

8.00 

0.71 

1.59 

82.90 

9.00 

0.57 

1.27 

84.17 

10.00 

0.54 

1.21 

85.38 

11.00 

0.51 

1.14 

86.52 

12.00 

0.47 

1.04 

87.56 

13.00 

0.45 

1.01 

88.56 

14.00 

0.43 

0.95 

89.51 

15.00 

0.40 

0.88 

90.40 

16.00 

0.37 

0.83 

91.23 

17.00 

0.33 

0.73 

91.96 

18.00 

0.31 

0.68 

92.65 

19.00 

0.29 

0.63 

93.28 

20.00 

0.26 

0.58 

93.86 

21.00 

0.23 

0.50 

94.36 

22.00 

0.22 

0.48 

94.85 

23.00 

0.21 

0.48 

95.32 

24.00 

0.20 

0.44 

95.76 

25.00 

0.18 

0.40 

96.16 

26.00 

0.17 

0.38 

96.54 

27.00 

0.17 

0.37 

96.92 

28.00 

0.15 

0.33 

97.24 

29.00 

0.13 

0.29 

97.54 

30.00 

0.13 

0.28 

97.82 

31.00 

0.12 

0.26 

98.08 

32.00 

0.11 

0.25 

98.33 

33.00 

0.11 

0.25 

98.58 

34.00 

0.10 

0.23 

98.81 

35.00 

0.09 

0.21 

99.01 

36.00 

0.08 

0.17 

99.18 

37.00 

0.07 

0.16 

99.33 

38.00 

0.06 

0.13 

99.47 

39.00 

0.06 

0.12 

99.59 

40.00 

0.05 

0.11 

99.70 

41.00 

0.05 

0.10 

99.80 

42.00 

0.03 

0.07 

99.87 

43.00 

0.02 

0.06 

99.93 

44.00 

0.02 

0.04 

99.97 

45.00 

0.01 

0.03 

100.00 
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OVERALL  FACTOR  ANALYSIS  TOTAL  VARIANCE  EXPLAINED,  TIME  2 
(CONVERGED  IN  10  ITERATIONS) 


Initial  Eigenvalues  Rotation  of  Sums  of  Squares  Loadings 

Component  Total  %  of  Variance  Cumulative  %  Total  %  of  Variance  Cumulative  % 


1.00 


Wit] 


3.00 

4.00 

5.00 

6.00 

7.00 

8.00 

9.00 

10.00 

11.00 

12.00 

13.00 

14.00 

15.00 

16.00 

17.00 

18.00 

19.00 

20.00 

21.00 

22.00 

23.00 

24.00 

25.00 

26.00 

27.00 

28.00 

29.00 

30.00 

31.00 

32.00 

33.00 

34.00 

35.00 

36.00 

37.00 

38.00 

39.00 

40.00 

41.00 

42.00 

43.00 

44.00 

45.00 


17.74 

39.43 

39.43 

10.48 

23.29 

23.29 

5.74 

12.76 

52.19 

6.77 

15.05 

38.34 

3.99 

8.87 

61.06 

4.72 

10.49 

48.83 

2.18 

4.84 

65.90 

4.66 

10.35 

59.18 

1.62 

3.60 

69.50 

3.91 

8.68 

67.87 

1.49 

3.31 

72.80 

1.93 

4.30 

72.16 

1.23 

2.74 

75.55 

1.52 

3.39 

75.55 

0.96 

2.14 

77.69 

0.82 

1.83 

79.51 

0.80 

1.79 

81.30 

0.62 

1.37 

82.67 

0.59 

1.31 

83.99 

0.52 

1.15 

85.14 

0.50 

1.12 

86.25 

0.46  1.03  87.28 

0.44  0.98  88.27 

0.40  0.89  89.16 

0.40  0.88  90.04 

0.37  0.82  90.86 

0.34  0.76  91.62 

0.33  0.73  92.35 

0.30  0.68  93.03 

0.29  0.66  93.68 

0.26  0.57  94.25 

0.22  0.49  94.75 

0.21  0.46  95.21 

0.20  0.45  95.66 

0.19  0.41  96.07 

0.17  0.39  96.46 

0.17  0.37  96.83 

0.15  0.34  97.17 

0.15  0.34  97.51 

0.14  0.31  97.82 

0.13  0.29  98.11 

0.12  0.26  98.37 

0.11  0.23  98.60 

0.10  0.22  98.83 

0.10  0.22  99.05 

0.09  0.20  99.25 

0.09  0.19  99.44 

0.07  0.16  99.60 

0.06  0.13  99.73 

0.05  0.11  99.84 

0.04  0.09  99.93 

0.03  0.07  100.00 


104 


Appendix  C:  Overall  Pearson  Correlations  Between  Selected  Variables 
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Computer  Self-Efficacy,  Time  1 
Perceived  Usefulness,  Time  1 
Perceived  Ease  of  Use,  Time  1 
Computer  Attitude,  Time  1 
Usage 

Computer  Self-Efficacy,  Time  2 
Computer  Self-Efficacy,  Time  2 
Perceived  Ease  of  Use,  Time  2 
Computer  Attitude,  Time  2 
Beginning  of  Course  Self-Rated  Performance 
End  of  Course  Self-Rated  Performance 


1.00  0.31**  0.46**  0.27**  0.20**  0.48**  0.21**  0  31**  0.25**  0.51**  0.42** 

1.00  0.54**  0.68**  0.21**  0.16**  0.54**  0.26**  0.40**  0.30**  0.20** 

1.00  0.57**  0.15**  0.32**  0.36**  0.45**  0.43**  0.42  0.40** 

1.00  0.07  0.22**  0.49**  0.33**  0.57**  0.31**  0.27** 

1.00  0.09  0.09  0.08  0.08  0.26**  0.18** 

1.00  0.40**  0.40**  0.41**  0.35**  0.45** 

1.00  0.51**  0.60**  0.26**  0.37** 

1.00  0.64**  0.28**  0.49** 

1.00  0.19**  0.43** 

1.00  0.56** 

1.00 


*  Correlation  is  significant  at  the  .05  level  (2-tailed) 
**  Correlation  is  significant  at  the  .01  level  (2-tailed) 
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Expected  Cum  Prob 


Appendix  D:  Self-Rated  Performance  Linear  Regression  Models 


DEPENDENT  VARIABLE:  Beginning  Of  Course  Performance  (Time  1) 
INDEPENDENT  VARIABLES:  Beginning  Of  Course  Measures  (Time  1) 


Dependent  Variable 


Beginning  of  Course  Self-Rated  Performance 


Std.  Dev  =  .99 
Mean  =  0.00 
N  =  197.00 


.  0.025.50.75  1  -01 .21 .51 .72.02.22.5 

2.22.01.71.51.21.075.50.250  0  5  0  5  0  5  0 

5  0  5  0  5  0 

Regression  Standardized  Residual 


Normal  P-P  Plot:  Standardized  Residual 
Beginning  of  Course  Self-Rated  Performance 


Observed  Cum  Prob 
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Scatterplot 

Beginning  of  Course  Self-Rated  Performance 


Beginning  of  Course  Self-Rated  Performance 
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Expected  Cum  Prob 


DEPENDENT  VARIABLE:  End  Of  Course  Performance  (Time  2) 
INDEPENDENT  VARIABLES:  Beginning  Of  Course  Measures  (Time  1) 


Dependent  Variable 

End  of  Course  Self-Rated  Performance 


oooooooooo 
Regression  Standardized  Residual 


Normal  P-P  Plot:  Standardized  Residual 
End  of  Course  Self-Rated  Performance 


Observed  Cum  Prob 
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Scatterplot 


End  of  Course  Self-Rated  Performance 


0  1  2  3  4  5  6 


End  of  Course  Self-Rated  Performance 
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Q  mm  rm 


Expected  Cum  Prob 


DEPENDENT  VARIABLE:  End  Of  Course  Performance  (Time  2) 
INDEPENDENT  VARIABLES:  End  Of  Course  Measures  (Time  2) 


Dependent  Variable 

End  of  Course  Self-Rated  Performance 


ooooo 
Regression  Standardized  Residual 


Normal  P-P  Plot:  Standardized  Residual 
End  of  Course  Self-Rated  Performance 


Observed  Cum  Prob 
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Regression  Standardized  Residual 


Scatterplot 

End  of  Course  Self-Rated  Performance 


□ 


-3 

0  1  2  3  4  5  6 

End  of  Course  Self-Rated  Performance 
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ected  Cum  Prob 


DEPENDENT  VARIABLE:  On-The-Job  Performance  (Time  3) 
INDEPENDENT  VARIABLES:  Beginning  Of  Course  Measures  (Time  1) 


Dependent  Variable 
On-the-Job  Self-Rated  Performance 


Regression  Standardized  Residual 

Normal  P-P  Plot:  Standardized  Residual 
On-the-Job  Self-Rated  Performance 


Observed  Cum  Prob 


Std.  Dev  =  .97 
Mean  =  .02 
N  =  47.00 
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Scatterplot 


On-the-Job  Self-Rated  Performance 


SP  3 
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Expected  Cum  Prob 


DEPENDENT  VARIABLE:  On-The-Job  Performance  (Time  3) 
INDEPENDENT  VARIABLES:  End  Of  Course  Measures  (Time  2) 


Dependent  Variable 
On-the-Job  Self-Rated  Peformance 


Regression  Standardized  Residual 


Normal  P-P  Plot:  Standardized  Residual 
On-the-Job  Self-Rated  Performance 


Observed  Cum  Prob 


Std.  Dev  =  .98 
Mean  =  0.00 
N  =  45.00 
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Scatter  plot 

On-the-Job  Self-Rated  Performance 


On-the-Job  Self-Rated  Performance 
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Expected  Cum  Prob 


DEPENDENT  VARIABLE:  On-The-Job  Performance  (Time  3) 
INDEPENDENT  VARIABLES:  On-The-Job  Measures  (Time  3) 


Dependent  Variable 
On-the-Job  Self-Rated  Performance 


Std.  Dev  =  .98 
Mean  =  0.00 
N  =  46.00 


. 0.Q25.50.75  1.01.21.51.72.02.2 

2.22. 01.71.51.21.Q75. 50.250  0  5  0  5  0  5 

5  0  5  0  5  0 


Regression  Standardized  Residual 


Normal  P-P  Plot:  Standardized  Residual 
On-the-Job  Self-Rated  Performance 


Observed  Cum  Prob 
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Scatterplot 


On-the-Job  Self-Rated  Performance 


On-the-Job  Self-Rated 
Peformance 
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Appendix  E:  C2IPS  Surveys 
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AFIT  SURVEY  OF  C2IPS  TRAINING 
(SAP3/SYSAD) 


USAF  Survey  Control  Number: 
USAF  SCN  98-33 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  training  here  as 
well  as  work  experience  will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  used  to  establish  metrics  for 
C2IPS  training  course  assessment. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFI  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFI  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve 
C2IPS  operator  and  system  administrator  performance  in  the  field.  No  analysis  of 
individual  responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be 
permitted  to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be 
published.  No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 
survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherre@afit.af.mil 

AFIT/LAS  DSN:  785-7777  ext.  2136 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  41  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

1 .  IMPORTANT :  For  identification  purposes,  fill  in  the  last  4  digits  of  your  Social 
Security  number  in  the  rightmost  part  of  the  “IDENTIFICATION  NUMBER”  section 
on  the  answer  sheet. 

2.  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

3.  Erase  cleanly  any  responses  you  wish  to  change. 

4.  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

5.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

6.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  on  the  answer 
sheet  so  that  your  responses  will  be  anonymous. 

Each  answer  sheet  item  provides  10  response  items.  Some  questionnaire  items  may 
require  a  response  from  1-7  or  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate 
response. 
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The  following  items  ask  you  to  indicate  your  confidence  in  whether  you  could  use  C2IPS  under  a 
variety  of  conditions.  For  each  of  the  following  conditions,  indicate  your  confidence  about  your 
judgment  by  choosing  a  number  from  1  to  9,  where  1  indicates  “Not  at  ail  confident”,  5  indicates 
“Moderately  confident,”  and  9  indicates  “Totally  confident”. 

For  example,  consider  the  following  sample  item: 

I  COULD  COMPLETE  THE  JOB  USING  C2IPS... 

...if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go.  Not  at  all  123456789 
Totally 

Confident 

Confident 


I  could  perform  my  assignment/duties  using  C2IPS... 

1 .  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 

2.  if  I  had  never  used  a  system  like  C2IPS  before. 

3 .  if  I  had  only  the  C2IPS  user  manuals  for  reference. 

4.  if  I  had  seen  someone  using  C2IPS  before  trying  it  myself 

5.  if  I  could  call  someone  for  help  if  I  got  stuck. 

6.  if  someone  else  had  helped  me  get  started. 

7.  if  I  had  a  lot  of  time  to  complete  the  job  for  which  C2IPS  was  provided. 

8.  if  I  had  just  the  built-in  help  facility  for  assistance. 

9.  if  someone  showed  me  how  to  do  it  first. 

10.  if  I  had  used  similar  systems  like  C2IPS  to  do  the  same  job. 

The  following  items  assess  your  views  about  C2IPS.  The  items  are  in  the  form  of  a  short  statement 
accompanied  by  the  7-point  scale  shown  below.  Fill  in  the  circle  on  your  answer  sheet  which  most 
closely  represents  your  view.  For  example,  consider  the  following  question: 

My  interaction  with  C2IPS  is  understandable.  Strongly  1  2  3  4  5  6  7  Strongly 

Disagree  Agree 


The  scale  numbers  correspond  to  the  following  views: 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 
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1 1 .  Using  C2IPS  improves  my  performance  on  the  job. 

12.  Using  C2IPS  in  my  job  increases  my  productivity. 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 

13.  Using  C2IPS  enhances  my  effectiveness  in  my  job. 

14.  I  find  C2IPS  to  be  useful  in  my  job. 

15.  My  interaction  with  C2IPS  is  understandable. 

16.  Interacting  with  C2IPS  does  not  require  a  lot  of  mental  effort. 

1 7.  I  find  C2IPS  to  be  easy  to  use. 

18.  I  find  it  easy  to  get  C2IPS  to  do  what  I  want  it  to  do. 

The  next  four  questions  each  have  a  different  scale: 

19.  Using  C2IPS  is  a _ idea.  Bad  1  2  3  4  5  6  7 

Good 


20.  Using  C2IPS  is  a _ idea.  Foolish  1  2  3  4  5  6  7 

Wise 


21.  I _ the  idea  of  using  C2IPS.  Dislike  1  2  3  4  5  6  7 

Like 


22.  Using  C2IPS  is _ .  Unpleasant  1  2  3  4  5  6  7 

Pleasant 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  system  administrator  tasks.  Use  5-point 
scale  shown  below  to  rate  yourself  on  your  ability  to  perform  these  tasks. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

A  rating  of  "3"  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience  and  skill 
level. 

How  well  do  you... 

23.  ...configure  IPS  Local  Area  Network  components? 

23 .  ...  enter  operating  system  commands  for  basic  operations? 

24.  . .  .enter  operating  system  commands  to  manipulate  software  elements? 

25.  . .  .create  an  automated  command  procedure? 

23 .  ...  perform  disk  management  tasks? 

24.  . . .configure  network  operating  system  software  in  accordance  with  established  guidelines? 

25.  ...perform  internet  addressing? 

26.  . .  .perform  TCP/IP  procedures? 

27.  ...  configure  the  Local  Area  Network? 

28.  . .  .configure  the  Wide  Area  Network  (WAN)  subsystem  for  global  operation? 

29.  . .  .perform  VMS  system  backup  and  environment  modification? 

30.  . .  .perform  SMC  system  administration  tasks? 

31.  . .  .establish  node  communications  parameters? 

32.  . . .supervise  system  operations? 

33.  . . .execute  SQL  statements  to  retrieve  and  display  database  information? 

34.  . . .manage  IPS  database  integrity? 

35.  . .  .perform  C2IPS  security  implementation  procedures? 

36.  . .  .perform  C2  IPS  security  maintenance  procedures? 

37.  . .  .configure  IPS  application  software? 

38.  ...operate  IPS  application  software? 

39.  ...perform  fault  isolation? 
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40 .  ...  perform  restoration  functions? 

The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance.  Use  the 
5-point  rating  scale  outlined  below  to  rate  yourself  on  your  overall  performance  on  C2IPS. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

4 1 .  Overall,  how  do  you  rate  your  C2IPS  performance? 


AFIT  SURVEY  OF  C2IPS  TRAINING 
(RP3/SYSAD) 


USAF  Survey  Control  Number: 
USAF  SCN  98-33 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  work  experience 
will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  correlated  with  previous  surveys 
conducted  during  C2IPS  training  to  establish  metrics  for  C2IPS  training  course 
assessment  and  improve  C2IPS  performance  in  the  field. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFI 12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFI  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve  C2IPS 
operator  and  system  administrator  performance  in  the  field.  No  analysis  of  individual 
responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be  permitted 
to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be  published. 
No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 
survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherre@afit.af.mil 

AFIT/LAS  DSN:  785-7777  ext.  2136 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  23  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

1 .  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

2.  Erase  cleanly  any  responses  you  wish  to  change. 

3.  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

4.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

5.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  so  that  your 
responses  will  be  anonymous. 

Each  response  has  1 0  circles  (numbered  1  through  1 0).  Some  questionnaire  items  may 
require  a  response  from  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate  response. 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  system  administrator  tasks. 
Rate  this  individual  on  his/her  overall  ability  to  perform  these  C2IPS  tasks.  Take 
into  consideration  any  performance  changes  you  attribute  to  recent  C2IPS  training, 
previous  C2IPS  experience,  and  the  individual's  performance  on  the  tasks  when 
compared  with  individuals  of  similar  experience  and  skill  level.  If  the  individual  has 
been  recently  assigned,  use  your  best  judgment  in  assigning  your  ratings.  Use  the 
following  5-point  scale  to  record  your  ratings: 

(1)  Performs  well  below  average 

(2)  Performs  below  average 

(3)  Performs  average 

(4)  Performs  above  average 

(6)  Performs  well  above  average 

A  rating  of  "3"  should  indicate  the  individual's  performance  is  "in  the  middle"  of  those 
with  similar  experience  and  skill  level. 


How  well  does  the  individual. . . 

1 .  ...  configure  IPS  Local  Area  Network  components? 

2.  . .  .enter  operating  system  commands  for  basic  operations? 

3.  . .  .enter  operating  system  commands  to  manipulate  software  elements? 

4.  ...  create  an  automated  command  procedure? 

5 .  ...  perform  disk  management  tasks? 

6.  . .  .configure  network  operating  system  software  IAW  AMC/local  guidelines? 

7.  . .  .perform  internet  addressing? 

8.  . .  .perform  TCP/IP  procedures? 

9.  . .  .configure  the  Local  Area  Network? 

10.  . .  .configure  the  Wide  Area  Network  (WAN)  subsystem  for  global  operation? 

11.  . .  .perform  VMS  system  backup  and  environment  modification? 

12.  ...  perform  SMC  system  administration  tasks? 
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13.  ...establish  node  communications  parameters? 

(1)  Performs  well  below  average 

(2)  Performs  below  average 

(3)  Performs  average 

(4)  Performs  above  average 

(5)  Performs  well  above  average 

14.  ...  supervise  system  operations? 

15.  . .  .execute  SQL  statements  to  retrieve  and  display  database  information? 

16.  . .  .manage  IPS  database  integrity? 

17.  . .  .perform  C2IPS  security  implementation  procedures? 

18.  . .  .perform  C2  IPS  security  maintenance  procedures? 

19.  . .  .configure  IPS  application  software? 

20.  . .  .operate  IPS  application  software? 

21.  . .  .perform  fault  isolation? 

22.  . .  .perform  restoration  functions? 

The  following  question  asks  you  to  make  an  overall  assessment  of  the  individual's 
C2IPS  performance.  Use  the  5-point  rating  scale  described  below. 

(1)  Performs  well  below  average 

(2)  Performs  below  average 

(3)  Performs  average 

(4)  Performs  above  average 

(5)  Performs  well  above  average 

23.  Overall,  how  do  you  rate  the  individual's  C2IPS  performance? 


130 


AFIT  ON-THE  JOB  SURVEY 
OF  C2IPS  TRAINING 
(SAP3/OPS) 


USAF  Survey  Control  Number: 
USAF  SCN  98-33 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  training  here  as 
well  as  work  experience  will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  used  to  establish  metrics  for 
C2IPS  training  course  assessment. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFI  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFI  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve  C2IPS 
operator  and  system  administrator  performance  in  the  field.  No  analysis  of  individual 
responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be  permitted 
to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be  published. 
No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 
survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherre@afit.af.mil 

AFIT/LAS  DSN:  785-7777  ext.  2136 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  46  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

1 .  IMPORTANT :  For  identification  purposes,  fill  in  the  last  4  digits  of  your  Social 
Security  number  in  the  rightmost  part  of  the  “IDENTIFICATION  NUMBER”  section 
on  the  answer  sheet. 

2.  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

3.  Erase  cleanly  any  responses  you  wish  to  change. 

4.  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

5.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

6.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  on  the  answer 
sheet  so  that  your  responses  will  be  anonymous. 

Each  answer  sheet  item  provides  10  response  items.  Some  questionnaire  items  may 
require  a  response  from  1-7  or  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate 
response. 
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The  following  items  ask  you  to  indicate  your  confidence  in  whether  you  could  use  C2IPS  under  a 
variety  of  conditions.  For  each  of  the  following  conditions,  indicate  your  confidence  about  your 
judgment  by  choosing  a  number  from  1  to  9,  where  1  indicates  “Not  at  all  Confident”,  5  indicates 
“Moderately  Confident,”  and  9  indicates  “Totally  Confident”. 

For  example,  consider  the  following  sample  item: 

I  COULD  COMPLETE  THE  JOB  USING  C2IPS... 

...if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go.  Not  at  all  123456789 

Totally  Confident 

Confident 


I  could  perform  my  assignment/duties  using  C2IPS... 

1 .  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 

2.  if  I  had  never  used  a  system  like  C2IPS  before. 

3.  if  I  had  only  the  C2IPS  user  manuals  for  reference. 

4.  if  I  had  seen  someone  using  C2IPS  before  trying  it  myself. 

5.  if  I  could  call  someone  for  help  if  I  got  stuck. 

6.  if  someone  else  had  helped  me  get  started. 

7.  if  I  had  a  lot  of  time  to  complete  the  job  for  which  C2IPS  was  provided. 

8.  if  I  had  just  the  built-in  help  facility  for  assistance. 

9.  if  someone  showed  me  how  to  do  it  first. 

10.  if  I  had  used  similar  systems  like  C2IPS  to  do  the  same  job. 

The  following  items  assess  your  views  about  C2IPS.  The  items  are  in  the  form  of  a  short  statement 
accompanied  by  the  7-point  scale  shown  below.  Fill  in  the  circle  on  your  answer  sheet  which  most 
closely  represents  your  view.  For  example,  consider  the  following  question: 


My  interaction  with  C2IPS  is  understandable. 


Strongly 

Disagree 


2  3  4  5  6  7  Strongly 
Agree 


The  scale  numbers  correspond  to  the  following  views: 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 


1 1 .  Using  C2IPS  improves  my  performance  on  the  job. 
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12.  Using  C2IPS  in  my  job  increases  my  productivity. 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 

1 3 .  Using  C2IPS  enhances  my  effectiveness  in  my  job. 

14.  I  find  C2IPS  to  be  useful  in  my  job. 

15.  My  interaction  with  C2IPS  is  understandable. 

16.  Interacting  with  C2IPS  does  not  require  a  lot  of  mental  effort. 

1 7.  I  find  C2IPS  to  be  easy  to  use. 

18.  I  find  it  easy  to  get  C2IPS  to  do  what  I  want  it  to  do. 

The  next  four  questions  each  have  a  different  scale: 

19.  Using  C2IPS  is  a _ idea.  Bad  1  2  3  4  5  6  7 

Good 


20.  Using  C2IPS  is  a _ idea.  Foolish  1  2  3  4  5  6  7 

Wise 


21.  I _ the  idea  of  using  C2IPS.  Dislike  1  2  3  4  5  6  7 

Like 


22.  Using  C2IPS  is _ .  Unpleasant  1  2  3  4  5  6  7 

Pleasant 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  operator  tasks.  Use  the  5-point  scale  shown 
below  to  rate  yourself  on  your  ability  to  perform  these  tasks. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

A  rating  of  "3"  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience  and  skill 
level. 

- Using  C2IPS,  how  well  do  you... 

23.  ...demonstrate  sign-on/sign-off  procedures? 

24.  ...  interpret  the  mission  summary  screen? 

25.  ...interpret  the  inbound  mission  screen? 

26.  . . .interpret  the  outbound  mission  screen? 

27.  .. .interpret  the  mission  monitoring  screen? 

28.  . .  .interpret  the  mission  schedule  manage  screen? 

29.  ...  interpret  incoming  messages  to  review? 

30.  ...  interpret  diplomatic  clearance  information? 

31.  ...  create  and  transmit  an  airlift  advisory  message? 

32.  ...create  and  transmit  and  airlift  departure  message? 

33.  ...create  and  transmit  and  airlift  delay  message? 

34.  . .  .create  and  transmit  and  airlift  arrival  message? 

35.  ...interpret  a  diplomatic  clearance  status  message? 

36.  ...interpret  a  mission  schedule  message? 

37.  . .  .revise  a  single  mission? 

38.  ...  interpret  the  aircraft  control  and  status  screen? 

39.  ...  interpret  the  notice  to  review  screen? 

40.  ...  interpret  the  ramp  status  screen? 

41.  . .  .receive  and  transmit  the  airlift  schedule  request  message? 

42.  . .  .assign  and  update  sequence  of  events  (SOE)? 

43 .  ...  interpret  the  incomplete/recurring  message  screen? 
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44.  ...  interpret  the  outgoing  message  to  approve  screen? 

45.  . .  .interpret  the  single  mission  display  screen? 

The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance.  Use  the 
same  5-point  rating  scale  outlined  below  to  rate  yourself  on  your  overall  performance  on  C2IPS. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

46.  Overall,  how  do  you  rate  your  C2IPS  performance? 


AFIT  SURVEY  OF  C2IPS  TRAINING 
(RP3/OPS) 


USAF  Survey  Control  Number: 
USAF  SCN  98-33 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  work  experience 
will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  correlated  with  previous  surveys 
conducted  during  C2IPS  training  to  establish  metrics  for  C2IPS  training  course 
assessment  and  improve  C2IPS  performance  in  the  field. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses.  _ 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFI 12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFI  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve  C2IPS 
operator  and  system  administrator  performance  in  the  field.  No  analysis  of  individual 
responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be  permitted 
to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be  published. 
No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 

survey.  _ 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherre@afit.af.mil 

AFIT/LAS  DSN:  785-7777  ext.  2136 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  24  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

1 .  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

2.  Erase  cleanly  any  responses  you  wish  to  change. 

3 .  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

4.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

5.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  so  that  your 
responses  will  be  anonymous. 

Each  response  has  10  circles  (numbered  1  through  10).  Some  questionnaire  items  may 
require  a  response  from  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate  response. 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  operator  tasks.  Rate  this 
individual  on  his/her  overall  ability  to  perform  these  C2IPS  tasks.  Take  into 
consideration  any  performance  changes  you  attribute  to  recent  C2IPS  training, 
previous  C2IPS  experience,  and  the  individual's  performance  on  the  tasks  when 
compared  with  individuals  of  similar  experience  and  skill  level.  If  the  individual 
has  been  recently  assigned,  use  your  best  judgment  in  assigning  your  ratings.  Use 
the  following  5-point  scale  to  indicate  your  ratings: 

(1)  Performs  well  below  average 

(2)  Performs  below  average 

(3)  Performs  average 

(4)  Performs  above  average 

(5)  Performs  well  above  average 

A  rating  of  "3"  should  indicate  the  individual's  performance  is  "in  the  middle"  of  those 
with  similar  experience  and  skill  level. 


Using  C2IPS,  how  well  does  the  individual.. . 

1 .  ...  demonstrate  sign-on/sign-off  procedures? 

2.  . .  .interpret  the  mission  summary  screen? 

3 .  ...  interpret  the  inbound  mission  screen? 

4.  ...  interpret  the  outbound  mission  screen? 

5.  . .  .interpret  the  mission  monitoring  screen? 

6.  . .  .interpret  the  mission  schedule  manage  screen? 

7.  ...  interpret  incoming  messages  to  review? 

8.  ...  interpret  diplomatic  clearance  information? 

9.  ...  create  and  transmit  an  airlift  advisory  message? 

10.  . .  .create  and  transmit  and  airlift  departure  message? 

11.  . .  .create  and  transmit  and  airlift  delay  message? 

12.  . .  .create  and  transmit  and  airlift  arrival  message? 

13.  . .  .interpret  a  diplomatic  clearance  status  message? 
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(1)  Performs  well  below  average 

(2)  Performs  below  average 

(3)  Performs  average 

(4)  Performs  above  average 

(5)  Performs  well  above  average 

14.  ...  interpret  a  mission  schedule  message? 

15.  . .  .revise  a  single  mission? 

16.  ...  interpret  the  aircraft  control  and  status  screen? 

17.  . .  .interpret  the  notice  to  review  screen? 

18.  ...  interpret  the  ramp  status  screen? 

19.  . .  .receive  and  transmit  the  airlift  schedule  request  message? 

20.  . .  .assign  and  update  sequence  of  events  (SOE)? 

21.  ...  interpret  the  incomplete/recurring  message  screen? 

22.  ...  interpret  the  outgoing  message  to  approve  screen? 

23.  . .  .interpret  the  single  mission  display  screen? 

The  following  question  asks  you  to  make  an  overall  assessment  of  the  individual's 
C2IPS  performance.  Use  the  5-point  rating  scale  described  below. 

(1)  Performs  well  below  average 

(2)  Performs  below  average 

(3)  Performs  average 

(4)  Performs  above  average 

(5)  Performs  well  above  average 

24.  Overall,  how  do  you  rate  the  individual's  C2IPS  performance? 
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AFIT  END  OF  COURSE  SURVEY 
OF  C2IPS  TRAINING 
(SAP2/SYSAD) 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  training  here  as 
well  as  work  experience  will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  used  to  establish  metrics  for 
C2IPS  training  course  assessment. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFR  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFR  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve  C2IPS 
operator  and  system  administrator  performance  in  the  field.  No  analysis  of  individual 
responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be  permitted 
to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be  published. 
No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 
survey. 

Contact  Information  — 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherrer@afit.af.mil 

AFIT/LAS  DSN:  785-2910 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  68  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

1 .  IMPORTANT :  For  identification  purposes,  fill  in  the  last  4  digits  of  your  Social 
Security  number  in  the  rightmost  part  of  the  “IDENTIFICATION  NUMBER”  section 
on  the  answer  sheet. 

2.  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

3.  Erase  cleanly  any  responses  you  wish  to  change. 

4.  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

5.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

6.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  on  the  answer 
sheet  so  that  your  responses  will  be  anonymous. 

Each  answer  sheet  item  provides  10  response  items.  Some  questionnaire  items  may 
require  a  response  from  1-7  or  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate 
response. 
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The  following  items  ask  you  to  indicate  your  confidence  in  whether  you  could  use  C2IPS  under  a 
variety  of  conditions.  For  each  of  the  following  conditions,  indicate  your  confidence  about  your 
judgment  by  choosing  a  number  from  1  to  9,  where  1  indicates  “Not  at  all  Confident”,  5  indicates 
“Moderately  Confident,”  and  9  indicates  “Totally  Confident”. 

For  example,  consider  the  following  sample  item: 

I  COULD  COMPLETE  THE  JOB  USING  C2IPS... 


...  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 
Totally 

Confident 


Not  at  all  1 
Confident 


2  3  4  5  6  7  8  9 


I  could  perform  my  assignment/duties  using  C2IPS... 

1 .  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 

2.  if  I  had  never  used  a  system  like  C2IPS  before. 

3 .  if  I  had  only  the  C2IPS  user  manuals  for  reference. 

4.  if  I  had  seen  someone  using  C2IPS  before  trying  it  myself. 

5.  if  I  could  call  someone  for  help  if  I  got  stuck. 

6.  if  someone  else  had  helped  me  get  started. 

7.  if  I  had  a  lot  of  time  to  complete  the  job  for  which  C2IPS  was  provided. 

8.  if  I  had  just  the  built-in  help  facility  for  assistance. 

9.  if  someone  showed  me  how  to  do  it  first. 

10.  if  I  had  used  similar  systems  like  C2IPS  to  do  the  same  job. 

The  following  items  assess  your  views  about  C2IPS.  The  items  are  in  the  form  of  a  short  statement 
accompanied  by  a  scale.  Fill  in  the  circle  on  your  answer  sheet  which  most  closely  represents  your 
view.  For  example,  consider  the  following  question: 

My  interaction  with  C2IPS  is  clear  and  understandable.  Strongly  1  2  3  4  5  6  7  Strongly 

Disagree  Agree 


The  scale  numbers  correspond  to  the  following  views: 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 
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1 1 .  Using  C2IPS  improves  my  performance  on  the  job/in  the  course. 

12.  Using  C2IPS  in  my  job/in  the  course  increases  my  productivity. 
(TURN  PAGE-*) 
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( 1 )  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 

13.  Using  C2IPS  enhances  my  effectiveness  in  my  job/in  the  course. 

14.  I  find  C2IPS  to  be  useful  in  my  job/in  the  course. 

15.  My  interaction  with  C2IPS  is  clear  and  understandable. 

16.  Interacting  with  C2IPS  does  not  require  a  lot  of  mental  effort. 

1 7.  I  find  C2IPS  to  be  easy  to  use. 

18.1  find  it  easy  to  get  C2IPS  to  do  what  I  want  it  to  do. 

The  next  four  questions  each  have  a  different  scale: 

23.  Using  C2IPS  is  a _ idea.  Bad  1  2  3  4  5  6  7 

Good 


24.  Using  C2IPS  is  a _ idea.  Foolish  1  2  3  4  5  6  7 

Wise 


25.  I _ the  idea  of  using  C2IPS.  Dislike  1  2  3  4  5  6  7 

Like 


26.  Using  C2IPS  is _ .  Unpleasant  1  2  3  4  5  6  7 

Pleasant 


(TURN 

PAGE->) 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  system  administrator  tasks.  GIVEN  WHAT 
YOU  NOW  KNOW  ABOUT  C2IPS  AFTER  YOUR  TRAINING,  RE-RATE  YOUR  ABILITY  TO 
PERFORM  THESE  C2IPS  TASKS  A  T  THE  BEGINNING  OF  THIS  COURSE.  Use  the  5-point  scale 
shown  below  to  record  your  ratings: 

( 1 )  Performed  well  below  average 

(2)  Performed  below  average 

(3)  Performed  average 

(4)  Performed  above  average 

(5)  Performed  well  above  average 

A  rating  of  "3"  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience  and  skill 
level. 

At  the  beginning  of  this  course,  how  well  did  you... 

23.  ...configure  IPS  Local  Area  Network  components? 

24.  . .  .enter  operating  system  commands  for  basic  operations? 

25.  . .  .enter  operating  system  commands  to  manipulate  software  elements? 

26.  ...  create  an  automated  command  procedure? 

27.  . .  .perform  disk  management  tasks? 

28.  . . .configure  network  operating  system  software  in  accordance  with  established  guidelines? 

29.  . .  .perform  internet  addressing? 

30.  . .  .perform  TCP/IP  procedures? 

31.  ...  configure  the  Local  Area  Network? 

32.  ...configure  the  Wide  Area  Network  (WAN)  subsystem  for  global  operation? 

33.  . .  .perform  VMS  system  backup  and  environment  modification? 

34.  . .  .perform  SMC  system  administration  tasks? 

35.  . .  .establish  node  communications  parameters? 

36.  ...supervise  system  operations? 

37.  ...execute  SQL  statements  to  retrieve  and  display  database  information? 

38.  . . .manage  IPS  database  integrity? 

39.  . . .perform  C2IPS  security  implementation  procedures? 

40.  . .  .perform  C2  IPS  security  maintenance  procedures? 

41.  ...  configure  IPS  application  software? 

42.  . . .operate  IPS  application  software? 
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(TURN  PAGE-*) 


43.  . .  .perform  fault  isolation? 

(1)  Performed  well  below  average 

(2)  Performed  below  average 

(3)  Performed  average 

(4)  Performed  above  average 

(5)  Performed  well  above  average 

44.  . .  .perform  restoration  functions? 

The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance  AT  THE 
BEGINNING  OF  THS  COURSE.  Use  the  5-point  rating  scale  outlined  below. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

4 1 .  Overall,  how  do  you  rate  your  C2IPS  performance  at  the  beginning  of  the  course? 


(TURN  PAGE-*) 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  system  administrator  tasks.  NOW,  RATE 
YOURSELF  ON  YOUR  ABILITY  TO  PERFORM  THESE  C2IPS  TASKS  AFTER  COMPLETING 
THIS  COURSE.  Use  the  5-point  scale  shown  below  to  record  your  ratings: 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

A  rating  of  ”3’'  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience  and  skill 
level. 

After  completing  this  course,  how  well  do  you. . . 

46.  ...  configure  IPS  Local  Area  Network  components? 

47.  . .  .enter  operating  system  commands  for  basic  operations? 

48.  . . .enter  operating  system  commands  to  manipulate  software  elements? 

49.  ...  create  an  automated  command  procedure? 

50.  . .  .perform  disk  management  tasks? 

51.  . .  .configure  network  operating  system  software  in  accordance  with  established  guidelines? 

52.  . .  .perform  internet  addressing? 

53.  ...perform  TCP/IP  procedures? 

54.  . .  .configure  the  Local  Area  Network? 

55.  . .  .configure  the  Wide  Area  Network  (WAN)  subsystem  for  global  operation? 

56.  . . .perform  VMS  system  backup  and  environment  modification? 

57.  . . .perform  SMC  system  administration  tasks? 

58.  . . .establish  node  communications  parameters? 

59.  . . .supervise  system  operations? 

60.  . .  .execute  SQL  statements  to  retrieve  and  display  database  information? 

61.  . .  .manage  IPS  database  integrity? 

62.  . .  .perform  C2IPS  security  implementation  procedures? 

63.  ...perform  C2  IPS  security  maintenance  procedures? 

64.  ...  configure  IPS  application  software? 

65.  ...operate  IPS  application  software? 

66.  . .  .perform  fault  isolation?  (TURN  PAGE^) 
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(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

67.  . .  .perform  restoration  functions? 

The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance  AFTER 
COMPLETING  THIS  COURSE.  Use  the  5-point  rating  scale  outlined  below. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

68.  Overall,  how  do  you  rate  your  C2IPS  performance  after  completing  this  course? 


END  OF  SURVEY 

-  ENSURE  ALL  QUESTIONS  HAVE  BEEN  ANSWERED 
-  ENSURE  THE  LAST  4-DIGITS  OF  YOUR  SSN  ARE  FILLED  IN  ON  THE  ANSWER  SHEET 
-  TURN  IN  THE  ANSWER  SHEET  TO  YOUR  INSTRUCTOR 
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OF  C2IPS  TRAINING 
(SAP2/OPS) 


USAF  Survey  Control  Number: 
USAF  SCN  98-33 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  training  here  as 
well  as  work  experience  will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  used  to  establish  metrics  for 
C2IPS  training  course  assessment. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFI  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFI  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve  C2IPS 
operator  and  system  administrator  performance  in  the  field.  No  analysis  of  individual 
responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be  permitted 
to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be  published. 
No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 
survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherrer@afit.af.mil 

AFIT/LAS  DSN:  785-2910 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  70  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

1 .  IMPORTANT :  For  identification  purposes,  fill  in  the  last  4  digits  of  your  Social 
Security  number  in  the  rightmost  part  of  the  “IDENTIFICATION  NUMBER”  section 
on  the  answer  sheet. 

2.  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

3.  Erase  cleanly  any  responses  you  wish  to  change. 

4.  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

5.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

6.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  on  the  answer 
sheet  so  that  your  responses  will  be  anonymous. 

Each  answer  sheet  item  provides  1 0  response  items.  Some  questionnaire  items  may 
require  a  response  from  1-7  or  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate 
response. 
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The  following  items  ask  you  to  indicate  your  confidence  in  whether  you  could  use  C2IPS  under  a 
variety  of  conditions.  For  each  of  the  following  conditions,  indicate  your  confidence  about  your 
judgment  by  choosing  a  number  from  1  to  9,  where  1  indicates  “Not  at  all  Confident”,  5  indicates 
“Moderately  Confident,”  and  9  indicates  “Totally  Confident”. 

For  example,  consider  the  following  sample  item: 

I  COULD  COMPLETE  THE  JOB  USING  C2IPS... 


...  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 
Totally 

Confident 


Not  at  all  1 
Confident 


2  3  4  5  6  7  8  9 


I  could  perform  my  assignment/duties  using  C2IPS... 

1 .  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 

2.  if  I  had  never  used  a  system  like  C2IPS  before. 

3.  if  I  had  only  the  C2IPS  user  manuals  for  reference. 

4.  if  I  had  seen  someone  using  C2IPS  before  trying  it  myself. 

5.  if  I  could  call  someone  for  help  if  I  got  stuck. 

6.  if  someone  else  had  helped  me  get  started. 

7.  if  I  had  a  lot  of  time  to  complete  the  job  for  which  C2IPS  was  provided. 

8.  if  I  had  just  the  built-in  help  facility  for  assistance. 

9.  if  someone  showed  me  how  to  do  it  first. 

10.  if  I  had  used  similar  systems  like  C2IPS  to  do  the  same  job. 

The  following  items  assess  your  views  about  C2IPS.  The  items  are  in  the  form  of  a  short  statement 
accompanied  by  a  scale.  Fill  in  the  circle  on  your  answer  sheet  which  most  closely  represents  your 
view.  For  example,  consider  the  following  question: 

My  interaction  with  C2IPS  is  understandable.  Strongly  1  2  3  4  5  6  7  Strongly 

Disagree  Agree 


The  scale  numbers  correspond  to  the  following  views: 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 
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1 1 .  Using  C2IPS  improves  my  performance  on  the  job. 

12.  Using  C2IPS  in  my  job  increases  my  productivity. 
(TURN  PAGE->) 
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(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 

13.  Using  C2IPS  enhances  my  effectiveness  in  my  job. 

14.  I  find  C2IPS  to  be  useful  in  my  job. 

15.  My  interaction  with  C2IPS  is  understandable. 

1 6.  Interacting  with  C2IPS  does  not  require  a  lot  of  mental  effort. 

17.  I  find  C2IPS  to  be  easy  to  use. 

18.  I  find  it  easy  to  get  C2IPS  to  do  what  I  want  it  to  do. 

The  next  four  questions  each  have  a  different  scale: 

19.  Using  C2IPS  is  a _ idea.  Bad  1  2  3  4  5  6  7 

Good 


20.  Using  C2IPS  is  a _ idea.  Foolish  1  2  3  4  5  6  7 

Wise 


21.  I _ the  idea  of  using  C2IPS.  Dislike  1  2  3  4  5  6  7 

Like 


22.  Using  C2IPS  is _ .  Unpleasant  1  2  3  4  5  6  7 

Pleasant 


(TURN 

PAGE-*) 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  operator  tasks.  GIVEN  WHAT  YOU  NOW 
KNOW  ABOUT  C2IPS  AFTER  YOUR  TRAINING,  RE-RATE  YOUR  ABILITY  TO  PERFORM 
THESE  C2IPS  TASKS  BEFORE  THE  BEGINNING  OF  THIS  COURSE.  Use  the  5-point  scale 
shown  below  to  record  your  ratings. 

(1)  Performed  well  below  average 

(2)  Performed  below  average 

(3)  Performed  average 

(4)  Performed  above  average 

(5)  Performed  well  above  average 

A  rating  of  "3"  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience  and  skill 
level. 

- Before  the  beginning  of  this  course,  how  well  did  you... 

23 .  ...  demonstrate  sign-on/sign-off  procedures? 

24.  ...  interpret  the  mission  summary  screen? 

25 .  ...  interpret  the  inbound  mission  screen? 

26.  ...  interpret  the  outbound  mission  screen? 

27.  . . .interpret  the  mission  monitoring  screen? 

28.  ...  interpret  the  mission  schedule  manage  screen? 

29.  ...  interpret  incoming  messages  to  review? 

30.  ...  interpret  diplomatic  clearance  information? 

31.  ...  create  and  transmit  an  airlift  advisory  message? 

32.  . .  .create  and  transmit  and  airlift  departure  message? 

33.  . . .create  and  transmit  and  airlift  delay  message? 

34.  ...create  and  transmit  and  airlift  arrival  message? 

35.  . . .interpret  a  diplomatic  clearance  status  message? 

36.  ...interpret  a  mission  schedule  message? 

37.  . .  .revise  a  single  mission? 

38.  . .  .interpret  the  aircraft  control  and  status  screen? 

39.  ...  interpret  the  notice  to  review  screen? 

40.  ...  interpret  the  ramp  status  screen? 

41 .  . .  .receive  and  transmit  the  airlift  schedule  request  message? 

42.  . .  .assign  and  update  sequence  of  events  (SOE)? 
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43 .  ...  interpret  the  incomplete/recurring  message  screen? 

(1)  Performed  well  below  average 

(2)  Performed  below  average 

(3)  Performed  average 

(4)  Performed  above  average 

(5)  Performed  well  above  average 

44.  ...  interpret  the  outgoing  message  to  approve  screen? 

45.  . . .interpret  the  single  mission  display  screen? 

The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance.  Use  the 
same  5-point  rating  scale  outlined  below  to  re-rate  yourself  on  your  overall  performance  on  C2IPS 
BEFORE  BEGINNING  THIS  COURSE. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

46.  Overall,  how  do  you  rate  your  C2IPS  performance  before  beginning  this  course? 


(TURN  PAGE->) 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  operator  tasks.  NOW,  RATE  YOURSELF 
ON  YOUR  ABILITY  TO  PERFORM  THESE  C2IPS  TASKS  AFTER  COMPLETING  THIS 
COURSE.  Use  the  5-point  scale  shown  below  to  record  your  ratings: 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(6)  Perform  well  above  average 

A  rating  of  "3"  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience  and  skill 
level. 

After  completing  this  course,  how  well  do  you... 

47.  . .  .demonstrate  sign-on/sign-off  procedures? 

48.  . . .interpret  the  mission  summary  screen? 

49.  ...  interpret  the  inbound  mission  screen? 

50.  ...  interpret  the  outbound  mission  screen? 

51.  ...  interpret  the  mission  monitoring  screen? 

52.  . .  .interpret  the  mission  schedule  manage  screen? 

53.  ...  interpret  incoming  messages  to  review? 

54.  ...  interpret  diplomatic  clearance  information? 

55.  . .  .create  and  transmit  an  airlift  advisory  message? 

56.  . .  .create  and  transmit  and  airlift  departure  message? 

57.  . .  .create  and  transmit  and  airlift  delay  message? 

58.  . . .create  and  transmit  and  airlift  arrival  message? 

59.  ...  interpret  a  diplomatic  clearance  status  message? 

60.  ...  interpret  a  mission  schedule  message? 

61.  ...revise  a  single  mission? 

62 .  ...  interpret  the  aircraft  control  and  status  screen? 

63 .  ...  interpret  the  notice  to  review  screen? 

64.  ...  interpret  the  ramp  status  screen? 

65.  . .  .receive  and  transmit  the  airlift  schedule  request  message? 

66.  . .  .assign  and  update  sequence  of  events  (SOE)? 
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67.  ...  interpret  the  incomplete/recurring  message  screen? 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

68.  ...  interpret  the  outgoing  message  to  approve  screen? 

69.  ...  interpret  the  single  mission  display  screen? 

The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance  AFTER 
COMPLETING  THIS  COURSE.  Use  the  5-point  rating  scale  outlined  below. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

70.  Overall,  how  do  you  rate  your  C2IPS  performance  after  completing  this  course? 


END  OF  SURVEY 

-  ENSURE  ALL  QUESTIONS  HAVE  BEEN  ANSWERED 
-  ENSURE  THE  LAST  4-DIGITS  OF  YOUR  SSN  ARE  FILLED  IN  ON  THE  ANSWER  SHEET 
TURN  IN  THE  ANSWER  SHEET  TO  YOUR  INSTRUCTOR 
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AFIT  BEGINNING  OF  COURSE  SURVEY 
OF  C2IPS  TRAINING 
(SAP1/SYSAD) 


USAF  Survey  Control  Number: 
USAF  SCN  98-33 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  training  here  as 
well  as  work  experience  will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  used  to  establish  metrics  for 
C2IPS  training  course  assessment. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFR  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFR  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve  C2IPS 
operator  and  system  administrator  performance  in  the  field.  No  analysis  of  individual 
responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be  permitted 
to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be  published. 
No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 
survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherrer@afit.af.mil 

AFIT/LAS  DSN:  785-2910 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  53  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

1 .  Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

2.  IMPORTANT :  For  identification  purposes,  fill  in  the  last  4  digits  of  your  Social 
Security  number  in  the  rightmost  part  of  the  “IDENTIFICATION  NUMBER”  section 
on  the  answer  sheet. 

3.  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

4.  Erase  cleanly  any  responses  you  wish  to  change. 

5.  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

6.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

7.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  on  the  answer 
sheet  so  that  your  responses  will  be  anonymous. 

Each  answer  sheet  item  provides  10  response  items.  Some  questionnaire  items  may 
require  a  response  from  1-7  or  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate 
response. 
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The  following  items  ask  you  to  indicate  your  confidence  in  whether  you  could  use  C2IPS  under  a 
variety  of  conditions.  For  each  of  the  following  conditions,  indicate  your  confidence  about  your 
judgment  by  choosing  a  number  from  1  to  9,  where  1  indicates  “Not  at  all  confident”,  5  indicates 
“Moderately  confident,”  and  9  indicates  “Totally  confident”. 

For  example,  consider  the  following  sample  item: 

I  COULD  COMPLETE  THE  JOB  USING  C2IPS... 

If  there  was  no  one  to  tell  me  what  to  do  as  I  go.  Not  at  all  123456789  Totally 

Confident  Confident 


I  could  perform  my  assignment/duties  using  C2IPS... 

1 .  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 

2.  if  I  had  never  used  a  system  like  C2IPS  before. 

3.  if  I  had  only  the  C2IPS  user  manuals  for  reference. 

4.  if  I  had  seen  someone  using  C2IPS  before  trying  it  myself. 

5 .  if  I  could  call  someone  for  help  if  I  got  stuck. 

6.  if  someone  else  had  helped  me  get  started. 

7.  if  1  had  a  lot  of  time  to  complete  the  job  for  which  C2IPS  was  provided. 

8.  if  I  had  just  the  built-in  help  facility  for  assistance. 

9.  if  someone  showed  me  how  to  do  it  first. 

10.  if  1  had  used  similar  systems  like  C2IPS  to  do  the  same  job. 

The  following  items  assess  your  views  about  C2IPS.  The  items  are  in  the  form  of  a  short  statement 
accompanied  by  the  7-point  scale  shown  below.  Fill  in  the  circle  on  your  answer  sheet  which  most 
closely  represents  your  view.  For  example,  consider  the  following  question: 

My  interaction  with  C2IPS  is  understandable.  Strongly  1  2  3  4  5  6  7  Strongly 

Disagree  Agree 


The  scale  numbers  correspond  to  the  following  views: 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 


1 1 .  Using  C2IPS  improves  my  performance  on  the  job. 

12.  Using  C2IPS  in  my  job  increases  my  productivity.  (TURN  PAGE-») 
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(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 

13.  Using  C2IPS  enhances  my  effectiveness  in  my  job. 

14.  I  find  C2IPS  to  be  useful  in  my  job. 

15.  My  interaction  with  C2IPS  is  understandable. 

1 6.  Interacting  with  C2IPS  does  not  require  a  lot  of  mental  effort. 

17.  I  find  C2IPS  to  be  easy  to  use. 

1 8.  I  find  it  easy  to  get  C2IPS  to  do  what  I  want  it  to  do. 


The  next  four  questions  each  have  a  different  scale: 

19.  Using  C2IPS  is  a _ idea.  Bad  1  2  3  4  5  6  7 

Good 


20.  Using  C2IPS  is  a _ idea.  Foolish  1  2  3  4  5  6  7 

Wise 


21.  1 _ the  idea  of  using  C2IPS.  Dislike  1  2  3  4  5  6  7 

Like 


22.  Using  C2IPS  is 
Pleasant 


Unpleasant  1  2  3  4  5  6  7 


(TURN 

PAGE-*) 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  system  administrator  tasks.  Use  the  5-point 
scale  shown  below  to  rate  yourself  on  your  ability  to  perform  these  tasks. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

A  rating  of  "3M  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience/skill  level. 
Using  C2IPS,  how  well  do  you. . . 

23 .  ...  configure  IPS  Local  Area  Network  components? 

24.  . .  .enter  operating  system  commands  for  basic  operations? 

25 .  ...  enter  operating  system  commands  to  manipulate  software  elements? 

26.  ...  create  an  automated  command  procedure? 

27.  . .  .perform  disk  management  tasks? 

28.  . . .configure  network  operating  system  software  in  accordance  with  established  guidelines? 

29.  . .  .perform  internet  addressing? 

30.  . .  .perform  TCP/IP  procedures? 

31.  ...  configure  the  Local  Area  Network? 

32.  . .  .configure  the  Wide  Area  Network  (WAN)  subsystem  for  global  operation? 

33 .  . .  .perform  VMS  system  backup  and  environment  modification? 

34.  . .  .perform  SMC  system  administration  tasks? 

35.  . . .establish  node  communications  parameters? 

36.  ...supervise  system  operations? 

37.  . .  .execute  SQL  statements  to  retrieve  and  display  database  information? 

38.  . . .manage  IPS  database  integrity? 

39.  . .  .perform  C2IPS  security  implementation  procedures? 

40.  . .  .perform  C2  IPS  security  maintenance  procedures? 

41.  ...  configure  IPS  application  software? 

42.  . . .operate  IPS  application  software? 

43.  . .  .perform  fault  isolation? 

44.  . .  .perform  restoration  functions?  (TURN  PAGE-*) 
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The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance.  Use  the 
5-point  rating  scale  outlined  below  to  rate  yourself  on  your  overall  performance  on  C2IPS. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(6)  Perform  above  average 

(7)  Perform  well  above  average 

45.  Overall,  how  do  you  rate  your  C2IPS  performance? 

The  following  items  focus  on  background  information. 

46.  Are  you  a(n): 

(1)  Enlisted 

(2)  Officer 

(3)  Wage  Grade  (WG)  Civilian  Employee 

(4)  General  Schedule  (GS)  Civilian  Employee 

(5)  Contractor 

(6)  Other 

47.  What  is  your  grade  level  (e.g.  E3,  02,  etc.)? 


(1) 

1-2 

(2) 

3-4 

(3) 

5-6 

(4) 

7-9 

(5) 

10-13 

(6) 

14-15 

(7) 

SES 

(8) 

N/A 

48.  If  you  are  enlisted,  what  is  your  skill  level?  (If  you  are  not  enlisted  answer  N/A) 

(1)  3-level 

(2)  5-level 

(3)  7-level 

(4)  N/A 

49.  What  is  your  age? 

(1)  Under  20  years 

(2)  20-30 

(3)  31-40 

(4)  41-50 

(5)  51-60 

(6)  Older  than  60  years 

50.  What  type  of  organization  do  you  work  in? 

(1)  Wing/Base 

(2)  Numbered  Air  Force 

(3)  MAJCOM  staff 

(4)  Other 

(TURN  PAGE-») 
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5 1 .  What  is  your  gender? 

(1)  Male 

(2)  Female 

52.  On  average,  how  much  time  during  the  day  do  you  spend  on  C2IPS? 

(1)  None 

(2)  Less  than  1  hour 

(3)  1-3  hours 

(4)  4-6  hours 

(5)  7-8  hours 

(6)  More  than  8  hours 

53.  How  much  previous  experience  do  you  have  with  C2IPS  in  your  job? 

(1)  None 

(2)  1-6  months 

(3)  7-12  months 

(4)  More  than  12  months 


END  OF  SURVEY 

-  ENSURE  ALL  QUESTIONS  HAVE  BEEN  ANSWERED 
ENSURE  THE  LAST  4-DIGITS  OF  YOUR  SSN  ARE  FILLED  IN  ON  THE  ANSWER  SHEET 
-  TURN  IN  THE  ANSWER  SHEET  TO  YOUR  INSTRUCTOR 
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AFIT  BEGINNING  OF  COURSE  SURVEY 
OF  C2IPS  TRAINING 
(SAP1/OPS) 


USAF  Survey  Control  Number: 
USAF  SCN  98-33 


DEPARTMENT  OF  THE  AIR  FORCE 
AIR  UNIVERSITY  (AETC) 

AIR  FORCE  INSTITUTE  OF  TECHNOLOGY 
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Information 

About  this  Research  Study 

Thank  you  for  agreeing  to  participate  in  this  research  study.  Your  training  here  as 
well  as  work  experience  will  make  an  important  contribution  to  the  goals  of  this  study. 

Description  of  the  study:  The  purpose  of  this  study  is  to  measure  the  impact  of  C2IPS 
training. 

How  your  responses  will  be  used:  The  results  will  be  used  to  establish  metrics  for 
C2IPS  training  course  assessment. 

Confidentiality  of  your  responses:  This  information  is  being  collected  for  research 
purposes  only.  No  one  in  your  unit,  base,  or  MAJCOM  will  ever  see  your  individual 
responses. 


PRIVACY  ACT  STATEMENT 

In  accordance  with  AFI  12-35,  paragraph  8,  the  following  information  is  provided  as 
required  by  the  Privacy  Act  of  1974. 

Authority:  10  U.S.C.  8012,  Secretary  of  the  Air  Force;  powers  and  duties;  delegation 
by;  implemented  by  AFI  30-23,  Air  Force  Personnel  Survey  Program. 

Purpose:  This  survey  is  being  conducted  to  collect  demographic,  affective,  learning,  and 
behavioral  data  regarding  C2IPS  training.  This  data  will  be  analyzed  to  determine  the 
interrelationships  of  these  data  and  their  impact  on  C2IPS  training. 

Routine  Use:  Future  C2IPS  training  can  draw  upon  techniques  proven  to  improve  C2IPS 
operator  and  system  administrator  performance  in  the  field.  No  analysis  of  individual 
responses  will  be  conducted  and  ONLY  members  of  the  research  team  will  be  permitted 
to  access  the  raw  data.  Reports  summarizing  trends  in  C2IPS  training  may  be  published. 
No  individual  will  be  identified  to  anyone  outside  of  the  research  team. 

Participation:  Participation  is  voluntary.  No  adverse  action  will  be  taken  against  any 
member  who  does  not  participate  in  this  survey  or  who  does  not  complete  any  part  of  this 
survey. 

Contact  Information 

If  you  have  any  questions,  please  feel  free  to  contact  me: 

Capt  Joe  Scherrer  email:  jscherrer@afit.af.mil 

AFIT/LAS  DSN:  785-2910 

2950  P  Street 

Wright-Patterson  AFB,  OH  45433-7765 
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INSTRUCTIONS 


This  questionnaire  contains  54  items  (individual  “questions”).  All  items  must  be 
answered  by  filling  in  the  appropriate  spaces  on  the  machine-scored  response  sheets 
provided.  If  for  any  item  you  do  not  find  a  response  that  fits  your  situation  exactly,  use 
the  one  that  is  closest  to  the  one  that  meets  your  situation. 

Please  use  a  “soft-lead”  (No.  2)  pencil,  and  observe  the  following: 

1 .  IMPORTANT :  For  identification  purposes,  fill  in  the  last  4  digits  of  your  Social 
Security  number  in  the  rightmost  part  of  the  “IDENTIFICATION  NUMBER”  section 
on  the  answer  sheet. 

2.  Answer  all  questions.  Make  heavy  black  marks  that  fill  in  the  numbered  circle  which 
represents  your  response. 

3.  Erase  cleanly  any  responses  you  wish  to  change. 

4.  Make  no  stray  markings  of  any  kind  on  the  response  sheet. 

5.  Do  not  staple,  fold,  or  tear  the  response  sheet. 

6.  Do  not  make  any  markings  on  the  questionnaire  sheet. 

You  have  been  provided  with  one  answer  sheet.  Do  NOT  fill  in  your  name  on  the  answer 
sheet  so  that  your  responses  will  be  anonymous. 

Each  answer  sheet  item  provides  1 0  response  items.  Some  questionnaire  items  may 
require  a  response  from  1-7  or  1-5  only.  Therefore,  be  careful  to  fill  in  the  appropriate 
response. 
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The  following  items  ask  you  to  indicate  your  confidence  in  whether  you  could  use  CUPS  under  a 
variety  of  conditions.  For  each  of  the  following  conditions,  indicate  your  confidence  about  your 
judgment  by  choosing  a  number  from  1  to  9,  where  1  indicates  “Not  at  all  Confident”,  5  indicates 
“Moderately  Confident,”  and  9  indicates  “Totally  Confident”. 

For  example,  consider  the  following  sample  item: 

I  COULD  COMPLETE  THE  JOB  USING  C2IPS... 

If  there  was  no  one  to  tell  me  what  to  do  as  I  go.  Not  at  all  123456789  Totally 

Confident  Confident 


I  could  perform  my  assignment/duties  using  C2IPS... 

1 .  if  there  was  no  one  around  to  tell  me  what  to  do  as  I  go. 

2.  if  I  had  never  used  a  system  like  C2IPS  before. 

3.  if  I  had  only  the  C2IPS  user  manuals  for  reference. 

4.  if  I  had  seen  someone  using  C2IPS  before  trying  it  myself 

5 .  if  I  could  call  someone  for  help  if  I  got  stuck. 

6.  if  someone  else  had  helped  me  get  started. 

7.  if  I  had  a  lot  of  time  to  complete  the  job  for  which  C2IPS  was  provided. 

8.  if  I  had  just  the  built-in  help  facility  for  assistance. 

9.  if  someone  showed  me  how  to  do  it  first. 

10.  if  I  had  used  similar  systems  like  C2IPS  to  do  the  same  job. 

The  following  items  assess  your  views  about  C2IPS.  The  items  are  in  the  form  of  a  short  statement 
accompanied  by  the  7-point  scale  shown  below.  Fill  in  the  circle  on  your  answer  sheet  which  most 
closely  represents  your  view.  For  example,  consider  the  following  question: 


My  interaction  with  C2IPS  is  understandable. 


Strongly 

Disagree 


2  3  4  5  6  7  Strongly 
Agree 


The  scale  numbers  correspond  to  the  following  views: 

(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 


1 1 .  Using  C2IPS  improves  my  performance  on  the  job. 

12.  Using  C2IPS  in  my  job  increases  my  productivity.  (TURN  PAGE->) 


174 


(1)  Strongly  Disagree 

(2)  Moderately  Disagree 

(3)  Somewhat  Disagree 

(4)  Neutral  (neither  Disagree  nor  Agree) 

(5)  Somewhat  Agree 

(6)  Moderately  Agree 

(7)  Strongly  Agree 

13.  Using  C2IPS  enhances  my  effectiveness  in  my  job. 

14.  I  find  C2IPS  to  be  useful  in  my  job. 

15.  My  interaction  with  C2IPS  is  understandable. 

1 6.  Interacting  with  C2IPS  does  not  require  a  lot  of  mental  effort. 

17.  I  find  C2IPS  to  be  easy  to  use. 

1 8.  1  find  it  easy  to  get  C2IPS  to  do  what  I  want  it  to  do. 


The  next  four  questions  each  have  a  different  scale: 


19.  Using  C2IPS  is  a 

idea. 

Bad  1 

20.  Using  C2IPS  is  a 

idea. 

Foolish  1 

21.  I  the  idea  of  using  C2IPS. 

Dislike  1 

22.  Using  C2IPS  is 

Unpleasant  1 

Pleasant 


2  3  4  5 

2  3  4  5 

2  3  4  5 

2  3  4  5 


6  7  Good 

6  7  Wise 

6  7  Like 

6  7 


(TURN 

PAGE->) 
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Following  is  a  generalized  list  of  standard  AMC  C2IPS  operator  tasks.  Use  the  5-point  scale  shown 
below  to  rate  yourself  on  your  ability  to  perform  these  tasks. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

A  rating  of  "3"  indicates  your  performance  is  "in  the  middle"  of  those  with  similar  experience/skill  level. 
Using  C2IPS,  how  well  do  you... 

23.  ...demonstrate  sign-on/sign-off procedures? 

24.  ...  interpret  the  mission  summary  screen? 

25.  ...  interpret  the  inbound  mission  screen? 

26.  ...  interpret  the  outbound  mission  screen? 

27.  ...  interpret  the  mission  monitoring  screen? 

28.  ...interpret  the  mission  schedule  manage  screen? 

29.  ...  interpret  incoming  messages  to  review? 

30.  . .  .interpret  diplomatic  clearance  information? 

31.  ...  create  and  transmit  an  airlift  advisory  message? 

32.  . .  .create  and  transmit  and  airlift  departure  message? 

33.  ...create  and  transmit  and  airlift  delay  message? 

34.  . .  .create  and  transmit  and  airlift  arrival  message? 

35.  ...interpret  a  diplomatic  clearance  status  message? 

36.  ...  interpret  a  mission  schedule  message? 

37.  ...revise  a  single  mission? 

38.  ...  interpret  the  aircraft  control  and  status  screen? 

39.  ...  interpret  the  notice  to  review  screen? 

40.  ...  interpret  the  ramp  status  screen? 

41.  . .  .receive  and  transmit  the  airlift  schedule  request  message? 

42.  ..  .assign  and  update  sequence  of  events  (SOE)? 

43 .  ...  interpret  the  incomplete/recurring  message  screen? 

44.  ...  interpret  the  outgoing  message  to  approve  screen?  (TURN  PAGE->) 
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(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

45.  ...  interpret  the  single  mission  display  screen? 

The  following  question  asks  you  to  make  an  overall  assessment  of  your  C2IPS  performance.  Use  the 
5-point  rating  scale  outlined  below  to  rate  yourself  on  your  overall  performance  on  C2IPS. 

(1)  Perform  well  below  average 

(2)  Perform  below  average 

(3)  Perform  average 

(4)  Perform  above  average 

(5)  Perform  well  above  average 

46.  Overall,  how  do  you  rate  your  C2IPS  performance? 

The  following  items  focus  on  background  information. 

47.  Are  you  a(n): 

(1)  Enlisted 

(2)  Officer 

(3)  Wage  Grade  (WG)  Civilian  Employee 

(4)  General  Schedule  (GS)  Civilian  Employee 

(5)  Contractor 

(6)  Other 

48.  What  is  your  grade  level  (e.g.  E3,  02,  etc.)? 


(1) 

1-2 

(2) 

3-4 

(3) 

5-6 

(4) 

7-9 

(5) 

10-13 

(6) 

14-15 

(7) 

SES 

(8) 

N/A 

49.  If  you  are  enlisted,  what  is  your  skill  level?  If  you  are  not  enlisted  answer  N/A 

(1)  3-level 

(2)  5-level 

(3)  7-level 
(5)  N/A 

50.  What  is  your  age? 

(1)  Under  20  years 

(2)  20-30 

(3)  31-40 

(4)  41-50 

(5)  51-60 

(6)  Older  than  60  years  (TURN  PAGE-*) 
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5 1 .  What  type  of  organization  do  you  work  in? 


(1)  Wing/Base 

(2)  Numbered  Air  Force 

(3)  MAJCOM  staff 

(4)  Other 

52.  What  is  your  gender? 

(1)  Male 

(2)  Female 

53.  On  average,  how  much  time  during  the  day  do  you  spend  on  C2IPS? 

(1)  None 

(2)  Less  than  1  hour 

(3)  1-3  hours 

(4)  4-6  hours 

(5)  7-8  hours 

(6)  More  than  8  hours 

54.  How  much  previous  experience  do  you  have  with  C2IPS  in  your  job? 

(1)  None 

(2)  1  -6  months 

(3)  7-12  months 

(4)  More  than  12  months 


END  OF  SURVEY 

-  ENSURE  ALL  QUESTIONS  HAVE  BEEN  ANSWERED 
-  ENSURE  THE  LAST  4-DIGITS  OF  YOUR  SSN  ARE  FILLED  IN  ON  THE  ANSWER  SHEET 
TURN  IN  THE  ANSWER  SHEET  TO  YOUR  INSTRUCTOR 
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